CN111752466B - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN111752466B CN111752466B CN201910239327.7A CN201910239327A CN111752466B CN 111752466 B CN111752466 B CN 111752466B CN 201910239327 A CN201910239327 A CN 201910239327A CN 111752466 B CN111752466 B CN 111752466B
- Authority
- CN
- China
- Prior art keywords
- data
- data block
- target
- segment
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000013500 data storage Methods 0.000 title claims abstract description 31
- 238000013507 mapping Methods 0.000 claims abstract description 89
- 238000007726 management method Methods 0.000 claims description 15
- 239000012634 fragment Substances 0.000 claims 2
- 230000008569 process Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000036541 health Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data storage method and device, wherein the method comprises the following steps: receiving target data; determining the starting position of an available storage space in a stripe and the size of a data block in the available storage space, wherein the stripe comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs; establishing a storage mapping relation between target data and data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space; and storing the target data into the target data block according to the storage mapping relation. In the application, the storage speed can be improved and the memory overhead can be reduced by the mode.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data storage method and apparatus.
Background
At present, in a storage system, after a data processing server receives data from a user, the data is generally cached in a memory, and then the data in the memory is written into a disk for storage.
However, this approach has the problem of writing and reading data multiple times, resulting in low storage speed and increased memory overhead for the data processing server.
Disclosure of Invention
In order to solve the above technical problems, embodiments of the present application provide a data storage method and apparatus, so as to achieve the purposes of improving storage speed and reducing memory overhead, and the technical scheme is as follows:
a data storage method, comprising:
receiving target data;
determining the starting position of an available storage space in a stripe and the size of a data block in the available storage space, wherein the stripe comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs;
establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space;
and storing the target data into a target data block according to the storage mapping relation.
Preferably, the establishing a memory mapping relationship between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space includes:
taking the initial position of the available storage space as a data block offset position, taking a data block to which the data block offset position belongs as an ith data block, and taking a region from the data block offset position to the last storage position of the first data block as a j-th segment, wherein i is an integer not smaller than 0, and j is an integer not smaller than 0;
Judging whether the length of the j-th segment is larger than the length of the target data;
if yes, establishing a mapping relation from the target data to the jth segment, and establishing a mapping relation from the target data to a kth backup segment, wherein the kth backup segment is an area which has the same length and different positions with the jth segment in the available storage space, and k is an integer not less than 0;
if not, taking the first data of the target data as a starting position, intercepting the data which accords with the length of the jth segment from the target data according to a data arrangement sequence, taking the data as jth intercepted part data, establishing a mapping relation from the jth intercepted part data to the jth segment, and establishing a mapping relation from the jth intercepted part data to the kth backup segment;
updating the data block offset position by the sum of the data block offset position and the length of the j-th segment, and judging whether the difference between the length of the target data and the data block offset position is larger than the length of an i+1th data block, wherein the i+1th data block is a data block different from the i-th data block;
if not, taking the (i+1) th data block as a (j+1) th segment, establishing a mapping relation from the rest part of data to the (j+1) th segment, and establishing a mapping relation from the rest part of data to a (k+1) th backup segment, wherein the rest part of data is the data except the first intercepting part of data in the target data;
If yes, taking the (i+1) th data block as a (j+1) th segment, taking the first data of the rest part data as a starting position, intercepting the data which accords with the length of the second segment from the rest part data according to a data arrangement sequence, taking the data as second intercepted part data, establishing a mapping relation from the second intercepted part data to the second segment, and establishing a mapping relation from the second intercepted part data to the second backup segment;
taking i+1 as i, j+1 as j, k+1 as k, and returning to the step of updating the data block offset position by the sum of the data block offset position and the length of the j-th segment until the difference between the length of the target data and the data block offset position is not greater than the length of the i+1-th data block.
Preferably, after storing the target data into the target data block according to the storage mapping relationship, the method further includes:
judging whether the target data block or the target backup data block is successfully written into the target data, wherein the target backup data block is a data block with the same size as the target data block and different positions;
if the target data block and the target backup data block are not successfully written into the target data, prompting to check the disk state;
And if one of the target data block and the target backup data block is not successfully written into the target data, stopping using the data block which is not successfully written into the target data.
Preferably, the method further comprises:
preferably, the method further comprises:
reading the target data from the target data block;
performing data coding on the target data, and distributing data blocks in a disk after data coding as backup data blocks;
and storing the data after the data encoding into the backup data block.
A data storage device, comprising:
the receiving module is used for receiving the target data;
the system comprises a determining module, a judging module and a judging module, wherein the determining module is used for determining the starting position of an available storage space in a strip and the size of a data block in the available storage space, the strip comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs;
the establishing module is used for establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space;
and the first storage module is used for storing the target data into a target data block according to the storage mapping relation.
Preferably, the establishing module is specifically configured to:
taking the initial position of the available storage space as a data block offset position, taking a data block to which the data block offset position belongs as an ith data block, and taking a region from the data block offset position to the last storage position of the first data block as a j-th segment, wherein i is an integer not smaller than 0, and j is an integer not smaller than 0;
judging whether the length of the j-th segment is larger than the length of the target data;
if yes, establishing a mapping relation from the target data to the jth segment, and establishing a mapping relation from the target data to a kth backup segment, wherein the kth backup segment is an area which has the same length and different positions with the jth segment in the available storage space, and k is an integer not less than 0;
if not, taking the first data of the target data as a starting position, intercepting the data which accords with the length of the jth segment from the target data according to a data arrangement sequence, taking the data as jth intercepted part data, establishing a mapping relation from the jth intercepted part data to the jth segment, and establishing a mapping relation from the jth intercepted part data to the kth backup segment;
Updating the data block offset position by the sum of the data block offset position and the length of the j-th segment, and judging whether the difference between the length of the target data and the data block offset position is larger than the length of an i+1th data block, wherein the i+1th data block is a data block different from the i-th data block;
if not, taking the (i+1) th data block as a (j+1) th segment, establishing a mapping relation from the rest part of data to the (j+1) th segment, and establishing a mapping relation from the rest part of data to a (k+1) th backup segment, wherein the rest part of data is the data except the first intercepting part of data in the target data;
if yes, taking the (i+1) th data block as a (j+1) th segment, taking the first data of the rest part data as a starting position, intercepting the data which accords with the length of the second segment from the rest part data according to a data arrangement sequence, taking the data as second intercepted part data, establishing a mapping relation from the second intercepted part data to the second segment, and establishing a mapping relation from the second intercepted part data to the second backup segment;
taking i+1 as i, j+1 as j, k+1 as k, and returning to the step of updating the data block offset position by the sum of the data block offset position and the length of the j-th segment until the difference between the length of the target data and the data block offset position is not greater than the length of the i+1-th data block.
Preferably, the apparatus further comprises:
the judging module is used for judging whether the target data block or the target backup data block is successfully written into the target data, wherein the target backup data block is a data block with the same size as the target data block and different positions;
the prompting module is used for prompting to check the state of the disk if the target data block and the target backup data block are not successfully written into the target data;
and the stop-use module is used for stopping using the data block which is not successfully written into the target data if one of the target data block and the target backup data block is not successfully written into the target data.
Preferably, the apparatus further comprises:
the writing module is used for sending the record whether the target data block or the backup data block is successfully written into the target data or not to the information management module;
the information management module is used for storing the record into the file associated with the strip.
Preferably, the apparatus further comprises:
the reading module is used for reading the target data from the target data block;
the data coding module is used for carrying out data coding on the target data, and distributing data blocks in the disk after data coding as backup data blocks;
And the second storage module is used for storing the data after the data encoding into the backup data block.
Compared with the prior art, the beneficial effects of this application are:
in the method, after receiving the target data, the starting position of the available storage space in the stripe and the size of the data block in the available storage space are determined, so that the storage mapping relation between the target data and the data block in the available storage space is established, the target data can be directly stored into the corresponding disk data block according to the storage mapping relation, only one-time writing is needed in the whole process, the number of times of data writing and reading is reduced, the storage speed is improved, and the data is not required to be cached in the memory, so that the memory overhead can be reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of a data storage method provided herein;
FIG. 2 is a block diagram of one strip provided herein;
FIG. 3 is a schematic diagram of a memory mapping relationship between target data and data blocks provided in the present application;
FIG. 4 is a flow chart of another data storage method provided herein;
FIG. 5 is a flow chart of yet another data storage method provided herein;
FIG. 6 is a flow chart of yet another data storage method provided herein;
fig. 7 is a schematic logic structure diagram of a data storage device provided in the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The embodiment of the application discloses a data storage method, which comprises the following steps: receiving target data; determining the starting position of an available storage space in a strip and the size of a data block in the available storage space; establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space; and storing the target data into a target data block according to the storage mapping relation. In the method, the storage speed can be improved, and the memory overhead can be reduced.
Next, a description will be given of a data storage method disclosed in an embodiment of the present application, where the data storage method is applied to a data processing server, referring to fig. 1, the data storage method may include:
step S11, receiving target data.
In this embodiment, when a user terminal initiates a data storage request, target data is received from the user terminal.
Step S12, determining the starting position of the available storage space in the strip and the size of the data block in the available storage space.
In this embodiment, the stripe may include at least one set of data blocks, where the data blocks in the set of data blocks belong to different disks. For example, the stripe includes a set of data blocks, where a set of data blocks includes 4 data blocks, respectively, 1, 2, 3, and 4, data block 1 is a data block in disk 1, data block 2 is a data block in disk 2, data block 3 is a data block in disk 3, and data block 4 is a data block in disk 4, where disks 1, 2, 3, and 4 are different from each other.
Of course, the stripe may also include multiple sets of data blocks. As shown in fig. 2, the stripe includes 3 sets of data blocks, namely, a first set of data blocks 1, 2, 3, 4, a second set of data blocks A, B, C, D, and a third set of data blocks a, b, c, d. The data blocks 1, 2, 3, and 4 may be used to store original data, the data block A, B, C, D may be used to store data for backing up the original data, and the data block a, b, c, d may also be used to store data for backing up the original data, so that the reliability of data storage may be improved.
And step S13, establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space.
After determining the starting position of the available storage space and the size of the data block in the available storage space in step S12, a memory mapping relationship between the target data and the data block in the available storage space is established if the length of the target data is known. The storage mapping relation between the target data and the data blocks in the available storage space is established, and the method can be understood as follows: and determining a data block used for storing target data in the available storage space, and establishing a corresponding relation between the data block used for storing the target data and the target data. If the size of the data block 1 is larger than the length of the target data, a memory mapping relationship between the target data and the data block 1 is established.
It should be noted that if one stripe cannot meet the storage requirement of the target data, a new stripe may be allocated after the storage mapping relationship between the data blocks in the stripe and the target data is established, and the storage mapping relationship between the remaining data in the target data and the data blocks in the new stripe is continuously established until the target data can be stored in the disk.
In this embodiment, a memory mapping relationship between the target data and the data blocks in the available storage space may be established with reference to a data writing process. The process of data writing may include:
(1) The user transmits the file to the data processing server, and the data processing server receives the data transmitted by the user. It is possible that the data transmitted by the user is smaller and it is also possible that the data transmitted is larger. Small data cannot be stored full of one stripe; some user data may just be stored full of one stripe; some data is more than one stripe of storage space in size.
(2) The data processing server firstly judges whether a strip is currently used or not, if so, the strip is directly used for storing data into a disk data block corresponding to the strip; if no stripe is currently allocated, a new stripe (e.g., a stripe comprising 4*3 data blocks) is applied to the information management module and the data is written to the new stripe. Here we determine the file size that has been saved at present by the variable fsize, which represents the current stripe, and size which represents the size of the amount of data that has been saved at the current stripe. The variable maxsz represents the size of the data amount stored by this stripe at most, and we use the stripe pattern of 4*3 here, then from the figure it is known that the effective original data amount stored by one stripe is 4 data blocks, and assuming that each data block is 4M, this data block stores at most 16M data, i.e. maxsz is 16M. When storing some data, the size of file fsize, strip, size inside the size strip, etc. is updated. When a stripe is full, the size is set to 0.
In detail, after the data processing server receives the data, firstly judging whether the variable size corresponding to the file is 0, if so, indicating that no stripe is allocated, the data processing server applies a new stripe to the information management server; if the value of the variable is greater than 0, the stripe is applied for, and the strip indicates the information of the stripe, including the data block corresponding to the stripe and the disk corresponding to the data block.
Assume that the size of the content of the charge file data is length. Then the size of the amount of data saved by the stripe is determined by length:
(1) If the size+length is less than or equal to maxsz, all data stored this time are completely stored in the current stripe.
(2) If the size+length is greater than maxsz, the amount of data of maxsz-size can be stored in the currently stored stripe, and the remaining length length=length- (maxsz-size) needs to be stored in the next stripe. If the next stripe is not stored, the stripe needs to be allocated again for storage. Until length is 0.
If the data block in the stripe is 4M, maxsz is 16M. There are 4 original data blocks, and size is the already stored data size. The size integer division 4M indicates the index of the starting data block that needs to be stored currently, for example, the size is 5M, and the size integer division 4M is equal to 1 by 5M. The size divided by 4M remainder indicates the current starting data block, and the already stored data size, e.g., the 5M divided by 4M remainder is equal to 1M, indicates that the current 2 nd data block has already stored 1M data content. The data block size 4M minus 1M indicates how much content the current data block also needs to store, here equal to 3M data. By the above calculation, the block number of the data block to be stored can be derived, the offset position of this block number, and this block can store the size of the data amount. The start position of the immediately following data block starts from 0 and the data size is 4M until this stripe is stored full. By the above calculation method, the starting position of the data block can be determined, and the size of the stored data can be determined.
And step S14, storing the target data into a target data block according to the storage mapping relation.
Storing the target data into the corresponding data blocks according to the storage mapping relation can be understood as: and storing the target data into a target data block in the magnetic disk according to the storage mapping relation. The target data block is a data block corresponding to the target data in the storage mapping relation.
In the method, after receiving the target data, the starting position of the available storage space in the stripe and the size of the data block in the available storage space are determined, so that the storage mapping relation between the target data and the data block in the available storage space is established, the target data can be directly stored into the corresponding disk data block according to the storage mapping relation, only one-time writing is needed in the whole process, the number of times of data writing and reading is reduced, the storage speed is improved, and the data is not required to be cached in the memory, so that the memory overhead can be reduced.
In another embodiment of the present application, a description of one implementation of step S13 in the foregoing embodiment may include:
a11, taking the initial position of the available storage space as a data block offset position, taking a data block to which the data block offset position belongs as an ith data block, taking a region from the data block offset position to the last storage position of the first data block as a jth segment, wherein i is an integer not smaller than 0, and j is an integer not smaller than 0.
A12, judging whether the length of the j-th segment is larger than the length of the target data.
If yes, executing the step A13; if not, step A14 is performed.
A13, establishing a mapping relation from the target data to the jth segment, and establishing a mapping relation from the target data to the kth backup segment, wherein the kth backup segment is an area which has the same length as the jth segment and is different in position in the available storage space, and k is an integer not less than 0.
A14, taking the first data of the target data as a starting position, intercepting the data which accords with the length of the jth segment from the target data according to a data arrangement sequence, taking the data as jth intercepted part data, establishing a mapping relation of the jth intercepted part data stored to the jth segment, and establishing a mapping relation of the jth intercepted part data stored to the kth backup segment.
A15, updating the data block offset position by the sum of the data block offset position and the length of the j-th segment, and judging whether the difference between the length of the target data and the data block offset position is larger than the length of an i+1th data block, wherein the i+1th data block is a data block different from the i-th data block.
If not, executing the step A16; if yes, go to step a17.
A16, taking the (i+1) th data block as a (j+1) th segment, establishing a mapping relation from the rest part of data to the (j+1) th segment, and establishing a mapping relation from the rest part of data to the (k+1) th backup segment, wherein the rest part of data is the data except the first intercepting part of data in the target data.
A17, taking the (i+1) th data block as a (j+1) th segment, taking the first data of the rest part data as a starting position, intercepting the data which accords with the length of the second segment from the rest part data according to a data arrangement sequence, taking the data as second intercepted part data, establishing a mapping relation of the second intercepted part data stored to the second segment, and establishing a mapping relation of the second intercepted part data stored to the second backup segment.
A18, taking i+1 as i, taking j+1 as j, taking k+1 as k, and returning to execute the step A15 until the difference between the length of the target data and the offset position of the data block is not greater than the length of the (i+1) th data block.
For example, referring to fig. 3, the offset position offset of the data block is set to 0, the offset+the data amount to be stored in the first data block is marked as segment 1, and if the available storage space of the first data block is greater than the length of the target data (denoted as length). The first data block stores data of size length, i.e. the data size of segment 1 is of size length.
If length is greater than the available memory space of the first data block, the size of the data amount of segment 1 is the available memory space of the first data block, and the offset is set to the size of offset+segment 1. If the length-offset is less than or equal to 4M (4M is the size of a certain data block), setting a storage area with the size of length-offset in the second data block as a segment 2; if length-offset is greater than 4M, set segment 2 as the storage space for the second data block, continue with the offset set to the size of offset + segment 2, and so on until the offset is equal to length or the stripe has been completely mapped one-to-one to the target data.
In another embodiment of the present application, another data storage method is extended, please refer to fig. 4, which may include:
step S21, receiving target data.
Step S22, determining the starting position of an available storage space in a stripe and the size of a data block in the available storage space, wherein the stripe comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs.
Step S23, according to the starting position of the available storage space and the size of the data block in the available storage space, a storage mapping relation between the target data and the data block in the available storage space is established.
And step S24, storing the target data into a target data block according to the storage mapping relation.
Steps S21 to S24 are identical to steps S11 to S14 in the foregoing embodiments, and the detailed procedure of steps S21 to S24 may be referred to the relevant description of steps S11 to S14, and will not be repeated here.
And S25, judging whether the target data block or the target backup data block is successfully written into the target data.
The target backup data block is a data block with the same size and different positions as the target data block. It will be appreciated that the size of the target backup data block is the same as the size of the target data block, but at a different location in the disk.
If the target data block and the target backup data block are not successfully written into the target data, executing step S26; if one of the target data block and the target backup data block is not successfully written into the target data, step S27 is executed.
Step S26, prompting to check the state of the disk.
Step S27, stopping using the data block which is not successfully written into the target data.
Preferably, the health of the target data block and the backup target data block may be set to mark whether the target data was successfully written. For example, the stripe includes target data blocks 1, 2, 3, 4, backup target data block A, B, C, D, and backup target data block a, b, c, d. Using 0 to indicate that writing of some or all data in the target data block or the backup target data block fails; 1 indicates that all written data in the target data block or the backup target data block is successful. Then the health degree of the three data blocks is the sequence xxx, the first x represents whether the storage of the target data block is all successful, the total success is 1, and otherwise, the health degree is 0; the middle x indicates whether the storage of the backup target data block A, B, C, D is all successful, and the storage is all successful is 1, otherwise, the storage is 0; the last x indicates whether the storage of the backup target data block a, b, c, d is all successful, with all successful being 1, otherwise 0. Then during the storage process, the best case is 111, indicating that all data blocks in the stripe were successfully stored; if a 0 occurs at any location, it indicates that there is a backup data write failure in the stripe, e.g., 100, 101, 011 … …, and a location of a 0 indicates that a backup storage in the stripe fails.
If all the stripes fail to be stored, a preset processing method (such as prompting to detect the state of the disk) is used; if one or two backups fail, then only the remaining two or one backups need to be stored when the user enters the data next time, and the damaged backup target data block does not store the data any more.
After the stripe is stored (which may not be full, but the user closes the file, which is also considered as the storage completion), the data recovery is performed on the data block according to the prompt information of the detected disk state
In another embodiment of the present application, another data storage method is described, referring to fig. 5, which may include:
step S31, receiving target data.
Step S32, determining the starting position of an available storage space in a stripe and the size of a data block in the available storage space, wherein the stripe comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs.
And step S33, establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space.
And step S34, storing the target data into a target data block according to the storage mapping relation.
And step S35, judging whether the target data block or the target backup data block is successfully written into the target data.
If the target data block and the target backup data block are not successfully written into the target data, executing step S36; if one of the target data block and the target backup data block is not successfully written into the target data, step S37 is executed.
Step S36, prompting to check the state of the disk.
Step S37, stopping using the data block which is not successfully written into the target data.
Steps S31 to S37 are identical to steps S21 to S27 in the foregoing embodiments, and the detailed procedure of steps S31 to S37 may be referred to the relevant description of steps S21 to S27, and will not be repeated here.
And step S38, the record of whether the target data block or the backup data block is successfully written into the target data is sent to an information management module.
The information management module is used for storing the record into the file associated with the strip, so that the storage condition of the data block corresponding to the strip can be conveniently inquired and managed.
In this embodiment, the information management module may be further configured to manage a file, a disk, a data block of the disk, a stripe formed by the data block of the disk, a relationship between the file and the stripe, and the like. Wherein the relationship of the file and the stripe is: when a file is written into the content data of the file, a stripe associated with the file needs to be associated with the file, and the data of the file is kept ordered, so that the stripe for storing the file is stored according to the content sequence stored in the file, and the operations such as data reading and the like are facilitated. Wherein the stripe comprises a plurality of data blocks, each from a different disk, all of the data blocks being identical in size. When each disk is associated with a service process and accesses the disk and a data block on the disk, the link address of the disk and the logical block number (block id) of the data block are used for accessing, wherein the link address is that the data processing server is connected with the disk, the block id is that the data block on the disk is numbered, the physical address of the data block in the disk can be determined through the number of the block, and the data on the disk is written or read through the physical address. The information management module manages a plurality of strips, and each strip comprises a data block of a plurality of disks; the data blocks of each disk are distributed in a plurality of stripes, and the data blocks in each stripe are not on the same disk.
In another embodiment of the present application, another data storage method is described, please refer to fig. 6, which may include:
step S41, receiving target data.
Step S42, determining the starting position of the available storage space in the stripe and the size of the data blocks in the available storage space, wherein the stripe comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs.
And step S43, establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space.
And step S44, storing the target data into a target data block according to the storage mapping relation.
Steps S41 to S44 are identical to steps S11 to S14 in the foregoing embodiments, and the detailed procedure of steps S41 to S44 may be referred to the relevant description of steps S11 to S14, and will not be repeated here.
Step S45, reading the target data from the target data block.
And step S46, encoding the target data, and distributing data blocks in a disk for the encoded data to serve as backup data blocks.
And the target data is subjected to data coding, so that the storage space can be saved.
And step S47, storing the data after the data encoding into the backup data block.
In this embodiment, the target data may be read from the target data block, or may be read from any one of the backup target data blocks of the stripe. After the target data is read, the target encodes the data. And then applying for a new stripe by using the information management module, storing the target data and the data after the data encoding into the data block corresponding to the new stripe, and deleting the target data in the data block corresponding to the original stripe. If the original stripe includes 4*3 data blocks, 4 data blocks are target data blocks, 4 data blocks are backup target data blocks, and the other 4 data blocks are backup target data blocks, after the target data is subjected to data encoding, a new stripe is allocated, the new stripe includes 4+2 data blocks, and the original target data is stored in disk data blocks corresponding to the 4 data blocks in the stripe; the encoded data is stored into 2 further data blocks.
Next, a description will be given of a data storage device provided in the present application, and the data storage device described below and the data storage method described above may be referred to correspondingly.
Referring to fig. 7, the data storage device includes: the device comprises a receiving module 11, a determining module 12, a setting-up module 13 and a first storage module 14.
A receiving module 11 for receiving target data;
a determining module 12, configured to determine a starting position of an available storage space in a stripe and a size of a data block in the available storage space, where the stripe includes at least one group of data blocks, and data blocks in the group of data blocks belong to different disks;
the establishing module 13 is configured to establish a storage mapping relationship between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space;
the first storage module 14 is configured to store the target data into a target data block according to the storage mapping relationship.
In this embodiment, the establishing module 13 may specifically be configured to:
taking the initial position of the available storage space as a data block offset position, taking a data block to which the data block offset position belongs as an ith data block, and taking a region from the data block offset position to the last storage position of the first data block as a j-th segment, wherein i is an integer not smaller than 0, and j is an integer not smaller than 0;
Judging whether the length of the j-th segment is larger than the length of the target data;
if yes, establishing a mapping relation from the target data to the jth segment, and establishing a mapping relation from the target data to a kth backup segment, wherein the kth backup segment is an area which has the same length and different positions with the jth segment in the available storage space, and k is an integer not less than 0;
if not, taking the first data of the target data as a starting position, intercepting the data which accords with the length of the jth segment from the target data according to a data arrangement sequence, taking the data as jth intercepted part data, establishing a mapping relation from the jth intercepted part data to the jth segment, and establishing a mapping relation from the jth intercepted part data to the kth backup segment;
updating the data block offset position by the sum of the data block offset position and the length of the j-th segment, and judging whether the difference between the length of the target data and the data block offset position is larger than the length of an i+1th data block, wherein the i+1th data block is a data block different from the i-th data block;
if not, taking the (i+1) th data block as a (j+1) th segment, establishing a mapping relation from the rest part of data to the (j+1) th segment, and establishing a mapping relation from the rest part of data to a (k+1) th backup segment, wherein the rest part of data is the data except the first intercepting part of data in the target data;
If yes, taking the (i+1) th data block as a (j+1) th segment, taking the first data of the rest part data as a starting position, intercepting the data which accords with the length of the second segment from the rest part data according to a data arrangement sequence, taking the data as second intercepted part data, establishing a mapping relation from the second intercepted part data to the second segment, and establishing a mapping relation from the second intercepted part data to the second backup segment;
taking i+1 as i, j+1 as j, k+1 as k, and returning to the step of updating the data block offset position by the sum of the data block offset position and the length of the j-th segment until the difference between the length of the target data and the data block offset position is not greater than the length of the i+1-th data block.
In this embodiment, the data storage device may further include:
the judging module is used for judging whether the target data block or the target backup data block is successfully written into the target data, wherein the target backup data block is a data block with the same size as the target data block and different positions;
the prompting module is used for prompting to check the state of the disk if the target data block and the target backup data block are not successfully written into the target data;
And the stop-use module is used for stopping using the data block which is not successfully written into the target data if one of the target data block and the target backup data block is not successfully written into the target data.
In this embodiment, the data storage device may further include:
the writing module is used for sending the record whether the target data block or the backup data block is successfully written into the target data or not to the information management module;
the information management module is used for storing the record into the file associated with the strip.
In this embodiment, the data storage device may further include:
the reading module is used for reading the target data from the target data block;
the data coding module is used for carrying out data coding on the target data, and distributing data blocks in the disk after data coding as backup data blocks;
and the second storage module is used for storing the data after the data encoding into the backup data block.
It should be noted that, in the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described as different from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other. For the apparatus class embodiments, the description is relatively simple as it is substantially similar to the method embodiments, and reference is made to the description of the method embodiments for relevant points.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present application.
From the above description of embodiments, it will be apparent to those skilled in the art that the present application may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in the embodiments or some parts of the embodiments of the present application.
The foregoing has described in detail a data storage method and apparatus provided herein, and specific examples have been presented herein to illustrate the principles and embodiments of the present application, the above examples being provided only to assist in understanding the method and core ideas of the present application; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (10)
1. A method of data storage, comprising:
receiving target data;
determining the starting position of an available storage space in a stripe and the size of a data block in the available storage space, wherein the stripe comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs;
establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space;
storing the target data into a target data block according to the storage mapping relation;
the establishing a memory mapping relationship between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space comprises the following steps:
Taking the initial position of the available storage space as a data block offset position, taking a data block to which the data block offset position belongs as an ith data block, and taking a region from the data block offset position to the last storage position of a first data block as a j-th segment, wherein i is an integer not smaller than 0, and j is an integer not smaller than 0;
judging whether the length of the j-th segment is larger than the length of the target data;
if yes, establishing a mapping relation from the target data to the j-th fragment;
if not, taking the first data of the target data as a starting position, intercepting the data which accords with the length of the jth segment from the target data according to a data arrangement sequence, taking the data as jth intercepted part data, and establishing a mapping relation from the jth intercepted part data to the jth segment;
updating the data block offset position by the sum of the data block offset position and the length of the j-th segment, and judging whether the difference between the length of the target data and the data block offset position is larger than the length of an i+1th data block, wherein the i+1th data block is a data block different from the i-th data block;
if not, taking the (i+1) th data block as a (j+1) th segment, and establishing a mapping relation between the rest part of data and the (j+1) th segment, wherein the rest part of data is the data except the first intercepting part of data in the target data;
If yes, taking the (i+1) th data block as a (j+1) th segment, taking the first data of the rest part data as a starting position, intercepting data which accords with the length of a second segment from the rest part data according to a data arrangement sequence, taking the data as second intercepted part data, and establishing a mapping relation of the second intercepted part data stored to the second segment;
taking i+1 as i, j+1 as j, k+1 as k, and returning to the step of updating the data block offset position by the sum of the data block offset position and the length of the j-th segment until the difference between the length of the target data and the data block offset position is not greater than the length of the i+1-th data block.
2. The method of claim 1, wherein the establishing a memory mapping relationship between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space further comprises:
establishing a mapping relation between the target data and a kth backup segment, wherein the kth backup segment is an area which has the same length as the jth segment and is different in position in the available storage space, and k is an integer not less than 0;
Establishing a mapping relation from the j-th intercepting part data to the k-th backup segment;
establishing a mapping relation from the rest data to the (k+1) th backup segment;
and establishing a mapping relation between the second intercepted part data and the second backup segment.
3. The method of claim 1, wherein after storing the target data in the target data block according to the memory mapping relationship, further comprising:
judging whether the target data block or the target backup data block is successfully written into the target data, wherein the target backup data block is a data block with the same size as the target data block and different positions;
if the target data block and the target backup data block are not successfully written into the target data, prompting to check the disk state;
and if one of the target data block and the target backup data block is not successfully written into the target data, stopping using the data block which is not successfully written into the target data.
4. A method according to claim 3, characterized in that the method further comprises:
the record of whether the target data block or the backup data block is successfully written into the target data is sent to an information management module;
The information management module is used for storing the record into the file associated with the strip.
5. The method according to claim 1, wherein the method further comprises:
reading the target data from the target data block;
performing data coding on the target data, and distributing data blocks in a disk after data coding as backup data blocks;
and storing the data after the data encoding into the backup data block.
6. A data storage device, comprising:
the receiving module is used for receiving the target data;
the system comprises a determining module, a judging module and a judging module, wherein the determining module is used for determining the starting position of an available storage space in a strip and the size of a data block in the available storage space, the strip comprises at least one group of data blocks, and the data blocks in the group of data blocks belong to different magnetic discs;
the establishing module is used for establishing a storage mapping relation between the target data and the data blocks in the available storage space according to the starting position of the available storage space and the size of the data blocks in the available storage space;
the first storage module is used for storing the target data into a target data block according to the storage mapping relation;
The building module is specifically configured to take a starting position of the available storage space as a data block offset position, take a data block to which the data block offset position belongs as an ith data block, and take a region from the data block offset position to a last storage position of a first data block as a jth segment, where i is an integer not less than 0, and j is an integer not less than 0;
judging whether the length of the j-th segment is larger than the length of the target data;
if yes, establishing a mapping relation from the target data to the j-th fragment;
if not, taking the first data of the target data as a starting position, intercepting the data which accords with the length of the jth segment from the target data according to a data arrangement sequence, taking the data as jth intercepted part data, and establishing a mapping relation from the jth intercepted part data to the jth segment;
updating the data block offset position by the sum of the data block offset position and the length of the j-th segment, and judging whether the difference between the length of the target data and the data block offset position is larger than the length of an i+1th data block, wherein the i+1th data block is a data block different from the i-th data block;
If not, taking the (i+1) th data block as a (j+1) th segment, and establishing a mapping relation between the rest part of data and the (j+1) th segment, wherein the rest part of data is the data except the first intercepting part of data in the target data;
if yes, taking the (i+1) th data block as a (j+1) th segment, taking the first data of the rest part data as a starting position, intercepting data which accords with the length of a second segment from the rest part data according to a data arrangement sequence, taking the data as second intercepted part data, and establishing a mapping relation of the second intercepted part data stored to the second segment;
taking i+1 as i, j+1 as j, k+1 as k, and returning to the step of updating the data block offset position by the sum of the data block offset position and the length of the j-th segment until the difference between the length of the target data and the data block offset position is not greater than the length of the i+1-th data block.
7. The apparatus of claim 6, wherein the means for establishing is further for:
establishing a mapping relation between the target data and a kth backup segment, wherein the kth backup segment is an area which has the same length as the jth segment and is different in position in the available storage space, and k is an integer not less than 0;
Establishing a mapping relation from the j-th intercepting part data to the k-th backup segment;
establishing a mapping relation from the rest data to the (k+1) th backup segment;
and establishing a mapping relation between the second intercepted part data and the second backup segment.
8. The apparatus of claim 6, wherein the apparatus further comprises:
the judging module is used for judging whether the target data block or the target backup data block is successfully written into the target data, wherein the target backup data block is a data block with the same size as the target data block and different positions;
the prompting module is used for prompting to check the state of the disk if the target data block and the target backup data block are not successfully written into the target data;
and the stop-use module is used for stopping using the data block which is not successfully written into the target data if one of the target data block and the target backup data block is not successfully written into the target data.
9. The apparatus of claim 8, wherein the apparatus further comprises:
the writing module is used for sending the record whether the target data block or the backup data block is successfully written into the target data or not to the information management module;
The information management module is used for storing the record into the file associated with the strip.
10. The apparatus of claim 6, wherein the apparatus further comprises:
the reading module is used for reading the target data from the target data block;
the coding module is used for coding the target data and distributing data blocks in a disk for the coded data to serve as backup data blocks;
and the second storage module is used for storing the encoded data into the backup data block.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910239327.7A CN111752466B (en) | 2019-03-27 | 2019-03-27 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910239327.7A CN111752466B (en) | 2019-03-27 | 2019-03-27 | Data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111752466A CN111752466A (en) | 2020-10-09 |
| CN111752466B true CN111752466B (en) | 2024-02-13 |
Family
ID=72671414
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910239327.7A Active CN111752466B (en) | 2019-03-27 | 2019-03-27 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111752466B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685232B (en) * | 2021-01-11 | 2022-03-01 | 河南大学 | Computer backup data monitoring method and system |
| CN112799595B (en) * | 2021-02-02 | 2023-06-23 | 联想(北京)有限公司 | Data processing method, device and storage medium |
| CN114063931B (en) * | 2021-11-26 | 2023-04-25 | 重庆科创职业学院 | Data storage method based on big data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2016162262A (en) * | 2015-03-03 | 2016-09-05 | 富士通株式会社 | Storage control device, control method and control program |
| CN107273048A (en) * | 2017-06-08 | 2017-10-20 | 浙江大华技术股份有限公司 | A kind of method for writing data and device |
| CN108572792A (en) * | 2017-06-13 | 2018-09-25 | 北京金山云网络技术有限公司 | Date storage method, device, electronic equipment and computer readable storage medium |
| JP6443572B1 (en) * | 2018-02-02 | 2018-12-26 | 富士通株式会社 | Storage control device, storage control method, and storage control program |
| CN109213430A (en) * | 2017-06-30 | 2019-01-15 | 伊姆西Ip控股有限责任公司 | Memory management method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109690681B (en) * | 2016-06-24 | 2021-08-31 | 华为技术有限公司 | Data processing method, storage device, solid state disk and storage system |
-
2019
- 2019-03-27 CN CN201910239327.7A patent/CN111752466B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2016162262A (en) * | 2015-03-03 | 2016-09-05 | 富士通株式会社 | Storage control device, control method and control program |
| CN107273048A (en) * | 2017-06-08 | 2017-10-20 | 浙江大华技术股份有限公司 | A kind of method for writing data and device |
| CN108572792A (en) * | 2017-06-13 | 2018-09-25 | 北京金山云网络技术有限公司 | Date storage method, device, electronic equipment and computer readable storage medium |
| CN109213430A (en) * | 2017-06-30 | 2019-01-15 | 伊姆西Ip控股有限责任公司 | Memory management method and system |
| JP6443572B1 (en) * | 2018-02-02 | 2018-12-26 | 富士通株式会社 | Storage control device, storage control method, and storage control program |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111752466A (en) | 2020-10-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111752466B (en) | Data storage method and device | |
| CN105718570B (en) | Data migration method and device for database | |
| CN104932953B (en) | A kind of data distributing method, date storage method, relevant apparatus and system | |
| CN110297603B (en) | Method and device for improving random writing performance based on solid state disk and computer equipment | |
| CN107679684A (en) | Declaration form distribution method, device, storage medium and computer equipment | |
| US20130290629A1 (en) | Storage system and storage apparatus | |
| CN102841849A (en) | Method and system for operating computerized memory | |
| CN103514216B (en) | Serial numbers based on Database Systems generate method | |
| EP2710477B1 (en) | Distributed caching and cache analysis | |
| CN111858189B (en) | Offline processing of storage disks | |
| CN110018996A (en) | A kind of the snapshot rollback method and relevant apparatus of distributed memory system | |
| US11010072B2 (en) | Data storage, distribution, reconstruction and recovery methods and devices, and data processing system | |
| CN112087497B (en) | Data synchronization method and device, electronic equipment and readable storage medium | |
| CN111352579B (en) | Data storage method and device | |
| CN111459399A (en) | Data writing method, data reading method and device | |
| CN111444046A (en) | Data recovery method and distributed data recovery system | |
| CN112765151A (en) | Random writing method and device based on distributed storage system and distributed storage system | |
| CN104407806B (en) | The amending method and device of RAID group hard disc information | |
| CN111427965B (en) | Management method and management system for water affair data | |
| GB2369902A (en) | Log storage management in a data processing system using key-pointing | |
| CN111506254B (en) | Distributed storage system and management method and device thereof | |
| CN106790635A (en) | Cookie information management method and server | |
| CN110489236A (en) | A kind of Internet of Things network interface card and its method for realizing FLASH load balancing | |
| CN114489504B (en) | SSD data management method and related components | |
| CN114546286B (en) | Method, system, storage medium and equipment for selecting members of homing group |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |