CN117240873B - Cloud storage system, data reading and writing method, device and storage medium - Google Patents
Cloud storage system, data reading and writing method, device and storage medium Download PDFInfo
- Publication number
- CN117240873B CN117240873B CN202311484532.2A CN202311484532A CN117240873B CN 117240873 B CN117240873 B CN 117240873B CN 202311484532 A CN202311484532 A CN 202311484532A CN 117240873 B CN117240873 B CN 117240873B
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- fragments
- available
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 781
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000002085 persistent effect Effects 0.000 claims abstract description 227
- 238000013500 data storage Methods 0.000 claims abstract description 129
- 230000004927 fusion Effects 0.000 claims abstract description 24
- 239000012634 fragment Substances 0.000 claims description 299
- 230000002159 abnormal effect Effects 0.000 claims description 125
- 230000002688 persistence Effects 0.000 claims description 103
- 238000007726 management method Methods 0.000 claims description 91
- 238000013507 mapping Methods 0.000 claims description 53
- 238000011084 recovery Methods 0.000 claims description 51
- 238000013467 fragmentation Methods 0.000 claims description 47
- 238000006062 fragmentation reaction Methods 0.000 claims description 47
- 238000004590 computer program Methods 0.000 claims description 21
- 238000012545 processing Methods 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 16
- 238000004891 communication Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000005856 abnormality Effects 0.000 description 6
- 230000003321 amplification Effects 0.000 description 4
- 238000003199 nucleic acid amplification method Methods 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000000977 initiatory effect Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- IUVCFHHAEHNCFT-INIZCTEOSA-N 2-[(1s)-1-[4-amino-3-(3-fluoro-4-propan-2-yloxyphenyl)pyrazolo[3,4-d]pyrimidin-1-yl]ethyl]-6-fluoro-3-(3-fluorophenyl)chromen-4-one Chemical compound C1=C(F)C(OC(C)C)=CC=C1C(C1=C(N)N=CN=C11)=NN1[C@@H](C)C1=C(C=2C=C(F)C=CC=2)C(=O)C2=CC(F)=CC=C2O1 IUVCFHHAEHNCFT-INIZCTEOSA-N 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Abstract
The embodiment of the application provides a cloud storage system, a data reading and writing method, data reading and writing equipment and a storage medium. In the embodiment of the application, a cloud storage disk with high availability across an available area is provided, data storage layout information of the cloud storage disk is generated according to disaster tolerance requirement information of the available area, and data read-write operation is performed on persistent storage nodes in the available area according to the data storage layout information. The data storage layout information can embody a redundancy storage mode of fusion of multiple copies and erasure codes adopted by the cloud storage disk, so that high availability of data storage across available areas is realized, and the data redundancy ratio is reduced; further, better read-write performance and the elasticity requirement of the available area are provided, the network bandwidth crossing the available area is reduced, and the storage cost is saved.
Description
Technical Field
The present disclosure relates to the field of cloud computing technologies, and in particular, to a cloud storage system, a data reading and writing method, a device, and a storage medium.
Background
Currently, mainstream cloud manufacturers promote storage services such as object storage, file storage, database and the like with disaster recovery capability. These storage services are deployed inside the same available area and implement disaster recovery using multiple copies or Erasure Coding (EC). With the development of cloud computing technology, realizing the disaster recovery of key services across available areas on the cloud becomes an increasingly interesting point for users on the cloud, and many users hope to quickly realize a service disaster recovery architecture by using a cloud hard disk.
Therefore, cloud manufacturers put forward a Yun Yingpan disaster recovery scheme for crossing available areas, namely storage clusters are respectively deployed in two available areas, when a user creates a cloud hard disk which is high in availability area, one cloud hard disk is respectively created in the storage clusters of the two available areas, the two cloud hard disks are mutually independent and mutually backup, and state synchronization of the two cloud hard disks is ensured through a master-slave synchronous replication technology. The Yun Yingpan disaster recovery scheme across the available area can ensure data consistency and realize disaster recovery across the available area, but the overall redundancy is higher.
Disclosure of Invention
Aspects of the application provide a cloud storage system, a data read-write method, a device and a storage medium, which are used for reducing storage redundancy ratio and realizing high-availability disaster recovery across available areas.
The embodiment of the application provides a cloud storage system, which comprises: the system comprises a storage service management node, a storage service node, a persistence management node and a plurality of persistence storage nodes, wherein the plurality of persistence storage nodes are distributed in a plurality of available areas; the storage service management node is used for creating a cloud storage disk crossing N available areas for the computing node, distributing storage service nodes for the cloud storage disk, enabling the distributed storage service nodes to respond to read-write requests initiated by the computing node to the cloud storage disk, wherein N is more than or equal to 2, and N is a positive integer; the cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation; the persistence management node is used for acquiring disaster recovery requirement information of the cross-available area of the cloud storage disk, generating data storage layout information required for reading and writing the cloud storage disk according to the disaster recovery requirement information of the cross-available area, wherein the data storage layout information is used for describing a storage mode of the coded slices and the multi-copy data slices in the persistence storage node of the cross-available area; and the storage service node is used for responding to a read-write request initiated by the computing node to the cloud storage disk and performing data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information.
The embodiment of the application also provides a data read-write method, which is suitable for the storage service management node in the cloud storage system, and comprises the following steps: creating a cloud storage disk which spans N available areas for a computing node, wherein N is more than or equal to 2, and N is a positive integer; the cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation; and distributing storage service nodes for the cloud storage disk, wherein the distributed storage service nodes perform data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information, and the data storage layout information is generated by the persistent management nodes in the cloud storage system according to the disaster recovery requirement information of the cross-available area of the cloud storage disk.
The embodiment of the application also provides a data read-write method, which is suitable for the persistent management node in the cloud storage system, and comprises the following steps: acquiring disaster recovery requirement information of a cloud storage disk in a cross-available area; the cloud storage disk spans N available areas and adopts a redundancy storage mode of fusion of multiple copies and erasure codes, wherein the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer; generating data storage layout information required for reading and writing the cloud storage disk according to the disaster recovery requirement information of the cross-available area, wherein the data storage layout information is used for describing the storage mode of the coded fragments and the multi-copy data fragments in the persistent storage nodes of the cross-available area; and providing the data storage layout information to the storage service nodes so that the distributed storage service nodes can perform data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information.
The embodiment of the application also provides a data read-write method, which is suitable for the storage service node in the cloud storage system, and comprises the following steps: receiving data storage layout information required by reading and writing a cloud storage disk and transmitted by a persistence management node in a cloud storage system, wherein the data storage layout information is generated by the persistence management node according to disaster tolerance requirement information of the cloud storage disk in a cross-available area; responding to a read-write request initiated by a computing node to a cloud storage disk, and performing data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information; the cloud storage disk spans N available areas, a redundancy storage mode of fusion of multiple copies and erasure codes is adopted, the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer.
The embodiment of the application also provides electronic equipment, which comprises: a memory and a processor; a memory for storing a computer program; and the processor is coupled with the memory and used for executing the computer program to realize each step in the data reading and writing method provided by the embodiment of the application.
The present application also provides a computer-readable storage medium storing a computer program, which when executed by a processor, causes the processor to implement the steps in the data read-write method provided in the embodiments of the present application.
In the embodiment of the application, a cloud storage disk with high availability across an available area is provided, data storage layout information of the cloud storage disk is generated according to disaster tolerance requirement information of the available area, and data read-write operation is performed on persistent storage nodes in the available area according to the data storage layout information. The data storage layout information can embody a redundancy storage mode of fusion of multiple copies and erasure codes adopted by the cloud storage disk, so that high availability of data storage across available areas is realized, and the data redundancy ratio is reduced; further, better read-write performance and the elasticity requirement of the available area are provided, the network bandwidth crossing the available area is reduced, and the storage cost is saved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
fig. 1 is a schematic structural diagram of a cloud storage system according to an exemplary embodiment of the present application;
fig. 2a is a schematic flow chart of a data read-write method according to an exemplary embodiment of the present application;
FIG. 2b is a schematic flow chart of another method for reading and writing data according to an exemplary embodiment of the present application;
FIG. 2c is a schematic flow chart of a further data read/write method according to an exemplary embodiment of the present application;
fig. 3a is a schematic structural diagram of a data read-write device according to an exemplary embodiment of the present application;
FIG. 3b is a schematic diagram of another data read-write device according to an exemplary embodiment of the present application;
FIG. 3c is a schematic diagram illustrating a structure of a data read-write device according to another exemplary embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.
In the present embodiment, the usable area (Availability Zone, AZ) refers to a physical area where power and a network are independent of each other within the same area. The network delay between cloud computing instances in the same available area is smaller, and the user access speed is faster. The area is a geographical area where the data center is located, and is generally divided according to a city where the data center is located. An area is typically defined by a plurality of available areas. In the same area, the available areas are communicated with each other through an intranet, and are connected by using a low-delay link; fault isolation can be realized among the available areas, namely if one available area fails, normal operation of other available areas cannot be affected. The available regions between the different regions are completely isolated.
With the development of cloud computing technology, realizing the disaster recovery of the cross-available area of key service on the cloud becomes an increasingly focused point of users on the cloud, and many users hope to quickly realize the service disaster recovery architecture by using a cloud hard disk and hope to realize the Yun Yingpan disaster recovery scheme of the cross-available area, so that the disaster recovery of the cross-available area can be realized while the data consistency is ensured, and a lower redundancy ratio is also needed.
In view of the above requirements, an embodiment of the present application provides a cloud storage disk with high availability across an available area, according to disaster tolerance requirement information across the available area, data storage layout information of the cloud storage disk is generated, and according to the data storage layout information, data read-write operation is performed on persistent storage nodes in the available area across the available area. The data storage layout information can embody a redundancy storage mode of fusion of multiple copies and erasure codes adopted by the cloud storage disk, and the data storage cross-available area high availability is realized through data synchronization and layout optimization, so that the data redundancy ratio is reduced. Further, better read-write performance and the elasticity requirement of the available area are provided, the network bandwidth crossing the available area is reduced, and the storage cost is saved.
One solution provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings.
Fig. 1 is a schematic structural diagram of a cloud storage system according to an exemplary embodiment of the present application. As shown in fig. 1, the system includes: a storage service management node 101, a storage service node 102, a persistence management node 103, and a plurality of persistence storage nodes 104. Optionally, as shown in fig. 1, the cloud storage system further includes: computing node 105 and cloud storage access node 106 carried on computing node 105.
In this embodiment, a cloud storage access node 106 is deployed on the computing node 105, where the cloud storage access node 106 may be deployed on the computing node 105 as a service process of a cloud storage system for providing access services to the cloud storage system to cloud computing instances (e.g., virtual machines or containers) running on the computing node 105. Specifically, cloud storage access node 106 may present cloud storage disks provided for it by the cloud storage system to cloud computing instances through virtualization techniques; a user may send an Input/Output (IO) read-write request to a cloud computing instance, which sends an IO read-write request to the cloud storage access node 106; the cloud storage access node 106 may receive an IO read-write request from the cloud computing instance to the cloud storage disk, send the IO read-write request to a backend storage cluster corresponding to the cloud storage disk, and return a response result to the cloud computing instance, such as a virtual machine or a container, after receiving the response result of the storage cluster. Wherein cloud storage access node 106 may interact with cloud computing instances through a virtualization manager on computing node 105 responsible for managing cloud computing instances. The virtualization manager may be a virtual machine monitor (Virtual Machine Monitor, VMM), also referred to as a Hypervisor. Alternatively, the Hypervisor or VMM may be embodied as a virtual operating system simulator (QEMU) or Vmware, but is not limited thereto.
In this embodiment, the backend cluster corresponding to the cloud storage disk includes: a service layer and a persistence layer, as shown in fig. 1. Wherein, the service layer includes: a storage service management node 101 and a storage service node 102. The storage service management node 101 is responsible for managing metadata of the cloud storage disk and managing and controlling the cloud storage disk. For example, cloud storage disks are created, deleted, mounted, and uninstalled, and the IO requests of the cloud storage disks are scheduled to be handled by which storage service node 102. For another example, a data synchronization and layout policy of the cloud storage disk may be specified, where the data synchronization and layout policy may embody a correspondence between the persistent file and the storage service node, e.g., the data synchronization and layout policy may be data storage layout information, see later for details. Storage service node 102 provides data interaction services for cloud storage disks, e.g., is responsible for handling IO requests from cloud storage access node 106, and optionally storage service node 102 may also be responsible for mapping management and switching scheduling of logical address information of the respective cloud storage disks to the respective persistent files. For example, the storage service node 102 stores the data to be stored in the IO request of the user to the persistence layer of the cloud storage disk, and records the corresponding data mapping relationship to facilitate subsequent reading.
Wherein the persistence layer comprises: a persistence management node 103 and a plurality of persistence storage nodes 104. The persistence management node 103 is responsible for managing metadata of a persistence layer of the cloud storage disk, optionally providing a management service for the persistence file, for example, being responsible for creating or deleting the persistence file, and for example, maintaining a correspondence relationship between the persistence file and the storage service node 102. Further, the persistence management node 103 can also persist data on the corresponding storage service node 102 according to the data synchronization and layout policy specified by the service layer. The storage service node 102 is responsible for responding to persistence operations and subsequent read accesses of the corresponding data. Further, the persistence layer can provide a software development kit (Software Development Kit, SDK) to the storage service node 102 of the service layer, so that the storage service node 102 can conveniently create, delete, read and write files of the persistence layer to complete the read and write operation of the user IO.
In the present embodiment, the implementation of the storage service management node 101, the storage service node 102, the persistent management node 103, and the plurality of persistent storage nodes 104 is also not limited. In fig. 1, the storage service management node 101 is a cloud hard disk management Server (BM), the storage service node 102 is a cloud hard disk data Server (BS), the persistent management node 103 is a metadata Server (Meta Server, MS), and the persistent storage node is a block Server (CS) for example, but is not limited thereto.
In the present embodiment, the deployment modes of the storage service management node 101, the storage service node 102, the persistent management node 103, and the plurality of persistent storage nodes 104 are not limited. In deployment, in order to meet the disaster tolerance of the available areas (namely, when any one of the available areas is completely unavailable, the cloud storage disk of the user can still normally perform read-write operation). Storage service management node 101 and persistent management node 103 are deployed in multiple Available areas and implement High Availability (HA), storage service node 102 and persistent storage node 104 are deployed in each Available area, and when an Available area is not Available, a new request is scheduled by storage service management node 101 and persistent management node 103 to normal storage service node 102 and persistent storage node 104. From the dimension of the available regions, each available region may include: a storage service management node 101, one or more storage service nodes 102, a persistence management node 103, and at least one persistence storage node 102; further still include: one or more computing nodes 105. In a specific deployment implementation, the storage service management node 101 and the storage service node 102 may be deployed together on one physical device (for example, the same storage device), or the storage service management node 101 and the storage service node 102 may be deployed on different physical devices. The persistent management node 103 may be co-deployed with the persistent storage node 104 on one physical device (e.g., the same storage device) or the persistent management node 103 and the persistent storage node 104 may be deployed on different physical devices.
Wherein, in fig. 1, a plurality of persistent storage nodes 102 are distributed in 3 available areas, and each available area includes 3 persistent storage nodes, a storage service management node, a storage service node and a persistent management node for illustration; the 3 available areas are: a usable area AZ1, a usable area AZ2, and a usable area AZ3; the usable area AZ1 includes: the storage service node availability zone AZ1 comprises: storage service node 102 (BS 1 and BS2, respectively), storage service management node 101 (e.g., BM 1), persistence management node 103 (e.g., MS 1), persistence storage node 104 (CS 1, CS2, and CS3, respectively); the usable area AZ2 further includes: storage service node 102 (BS 3 and BS4, respectively), storage service management node 101 (e.g., BM 2) and persistence management node 103 (e.g., MS 2), persistence storage node 104 (e.g., CS4, CS5, and CS 6); the usable area AZ3 further includes: storage service node 102 (BS 5 and BS6, respectively), storage service management node 101 (e.g., BM 3) and persistence management node 103 (e.g., MS 3), persistence storage node 104 (CS 7, CS8, and CS9, respectively). In fig. 1, the data reading and writing of the data plane is represented by a "solid line", the reading and writing service provided by the service node BS2 is illustrated by an example, the interaction of the metadata and the management information of the control plane is represented by a "dashed line", wherein the control of the BS by the BM1 and the interaction of the control information among the BM1, the BM2 and the BM3 are illustrated by an example, and the process of the other data planes and the control plane are not illustrated in fig. 1.
The detailed functions of the storage service management node 101, the storage service node 102, the persistence management node 103, and the persistence storage node 104 in the embodiment of the present application are described below.
In the embodiment, the storage service management node 101 is responsible for creating a cloud storage disk spanning N available areas for the computing node 105, where N is greater than or equal to 2 and N is a positive integer; on the other hand, the storage service management node 101 is responsible for allocating the storage service node 102 to the cloud storage disk, so that the allocated storage service node 102 responds to a read-write request initiated by the computing node 105 to the cloud storage disk. Wherein, the existing cloud storage disk can provide storage service for the computing nodes in the N available areas across the N available areas, and the backend persistent storage nodes 104 corresponding to the cloud storage disk can be distributed in the N available areas.
The cloud storage disk is used to provide storage services for cloud computing instances on the computing node, for example, to store various data generated by the cloud computing instances on the computing node 103. The cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple-copy data fragmentation. Wherein, the multi-copy means that the original data (such as the data to be stored) is copied into multiple copies for storage; the erasure code is a data protection method, dividing the original data into m data slices, coding the m data slices to obtain n coding slices, storing the n coding slices and the m data slices, and if any m slices (data slices or coding slices) can be obtained under the condition that a hard disk failure occurs, recovering the original data according to any m obtained slices. The redundant storage mode of multi-copy and erasure code fusion is a redundant storage strategy that original data (such as data to be stored) is encoded by erasure codes to obtain data fragments and encoded fragments, and the data fragments are subjected to multi-copy and cross-available-area storage. Alternatively, the coded slices may be stored in a single copy, preferably across the available regions. Of course, the code slicing may also employ multiple copy storage, where the redundancy ratio of multiple copy storage is higher than that of single copy storage.
In this embodiment, the cloud storage disk may be addressed by logical block addressing (Logical Block Addressing, LBA), where the entire storage space is divided into multiple LBA intervals, and a certain amount of data is stored in these LBA intervals. The size of the LBA interval is not limited, and may be, for example, 4GB, 8GB, 16GB, or the like. The LBA address information can be mapped with the physical address information of the persistent storage node so as to realize the reading and writing of the cloud storage disk.
Further, the storage service management node 101 may also maintain metadata of the cloud storage disk. For example, a tenant creates a cloud storage disk for a cloud computing instance on a computing node, and information such as a tenant identifier, a size of a required cloud storage disk, performance requirements and the like can be provided to the storage service management node 101, and the storage service management node 101 creates the cloud storage disk for the computing node of the tenant according to the needs of the tenant, and maintains metadata of the cloud storage disk. Wherein the metadata of the cloud storage disk may include, but is not limited to: disk identification (disk id), disk size (disk size), tenant identification (user), creation time (create time), number of times of unit reading and writing of the disk (disk Input/Output Operations Per Second capacity), disk throughput (disk throughput capacity), disk status (disk status), and the like may be expressed as (disk id, disk size, user, create time, disk IOPS capacity, disk throughput capacity, disk status).
Optionally, the storage service nodes 102 are plural, and the plural storage service nodes 102 are distributed in different available areas; the storage service management node 101 distributes the storage service nodes 102 to the cloud storage disk according to the available areas to which the storage service nodes 102 belong by adopting a nearby scheduling policy, and ensures that the storage service nodes 102 responding to the read-write request of the cloud storage disk and the cloud storage access node 106 are in the same available area as much as possible so as to improve the response speed.
In this embodiment, the persistence management node 103 may obtain disaster recovery requirement information of a cloud storage disk in a cross-available area, where the disaster recovery requirement information of the cross-available area is requirement information of availability of a tenant to the cloud storage disk, for example, the disaster recovery requirement information of the cross-available area may be information that allows several available areas to fail, allows several data fragments to fail, or allows several data fragments to fail when a set number of available areas are allowed to fail under the condition that normal data reading and writing of cloud computing instances on the computing node are ensured. And generating data storage layout information required for reading and writing the cloud storage disk according to the disaster recovery requirement information of the cross-available area, wherein the data storage layout information is used for describing the storage mode of the coded slices and the multi-copy data slices in the persistent storage node 104 of the cross-available area. That is, the data storage layout information embodies how the encoded shards are stored on the plurality of persistent storage nodes 104 of the N available areas and how the multi-copy data shards are stored on the plurality of persistent storage nodes 104 of the N available areas. For example, in the case where the available area is 3, the data slice is m, and the code slice is n, the data storage layout information may be: storing one copy in each of the 2 available areas, the copy comprising: m data fragments, n coding fragments are stored in three available areas in a scattered way; alternatively, the data storage layout information may also be: storing one copy in each of the 2 available areas, the copy comprising: m data slices, n code slices are stored in another available region and any available region containing copies.
In this embodiment, the storage service node 102 may acquire data storage layout information provided by the persistence management node from the persistence management node 103; in the case of a read-write request initiated by the computing node 105 to the cloud storage disk, responding to the read-write request initiated by the computing node 105 to the cloud storage disk, and performing data read-write operation on the persistent storage nodes 104 in the cross-available area according to the data storage layout information.
Alternatively, the storage service node 102 may obtain the data storage layout information from the persistence management node 103 when it first receives a write request initiated by the computing node 105 to the cloud storage disk. Alternatively, the storage service node 102 acquires data storage layout information from the persistence management node 103 at the time of power-on initialization.
In the embodiment of the application, a cloud storage disk with high availability across an available area is provided, data storage layout information of the cloud storage disk is generated according to disaster tolerance requirement information of the available area, and data read-write operation is performed on persistent storage nodes in the available area according to the data storage layout information. The data storage layout information can embody a redundancy storage mode of fusion of multiple copies and erasure codes adopted by the cloud storage disk, realizes high availability of data storage across available areas, and reduces the data redundancy ratio. Further, better read-write performance and the elasticity requirement of the available area are provided, the network bandwidth crossing the available area is reduced, and the storage cost is saved.
In an alternative embodiment, the disaster recovery requirement information across the available area includes: when M available areas in N available areas are abnormal, the tolerable abnormal slicing quantity f is that is to say, when the M available areas in the N available areas are abnormal and f slicing anomalies occur in non-abnormal available areas, the read-write operation of the cloud storage disk can be guaranteed. The abnormal slices may include data slices or code slices. Wherein M is more than or equal to 1 and less than N, f is more than or equal to 1, and M and f are positive integers. In addition, M, N and f are known values. For example, disaster recovery requirement information across the available area includes: when 1 available area in 3 available areas is abnormal, 2 abnormal fragments can be tolerated, namely, when 1 available area in 3 available areas is abnormal and 2 fragments are abnormal, normal reading and writing can be carried out. The data storage layout information specifically includes: the storage mapping relationship between p×m+n slices and K persistent storage nodes indicates that p×m+n slices are stored in K persistent storage nodes, where K persistent storage nodes are at least distributed in P available areas, for example, may be distributed in P available areas or may be distributed in p+l available areas, and L is a positive integer, preferably, l=1, which is not limited. Wherein, P is less than or equal to 2 and less than or equal to (N-1), N is more than or equal to f, P, K, m and N are positive integers, the "% represents multiplication operation, m and N are the number of data slices and coding slices in erasure codes respectively, P represents the number of copies of m data slices, and different copies of m data slices are located in different available areas.
In an alternative embodiment, the selection of the P available regions and the K persistent storage nodes is based on meeting the following disaster recovery requirements across the available regions:
a) And ensuring that the cross-available area of the cloud storage disk data read-write is high in availability. In the worst case of data exception, even if M (e.g., m=1) available areas are abnormal and f (e.g., f has a value of 1 or 2) data fragments are not available, normal read-write access to the cloud storage disk can be ensured.
b) When data is written into the cloud storage disk, the data flow of writing crossing the available area is ensured to be as small as possible. Meanwhile, the data reading can be completed in the local available area, the overall reading and writing performance is improved, and the network bandwidth requirement across the available area is reduced.
c) When the available area is abnormal or the fragmentation abnormality occurs, the read data flow rate across the available area is as small as possible.
d) The data redundancy ratio is relatively low, and the data redundancy ratio is close to the number of copies P.
For example, to ensure that the read data in both available areas of the user service disaster recovery is completed within the present available area, there may be 1 copy placed in the local available area, while to satisfy the available area disaster recovery, at least 1 copy is distributed in the other available area, and the data redundancy ratio is close to 2.
Based on the foregoing, an embodiment for generating data storage layout information required for reading and writing a cloud storage disk according to disaster tolerance requirement information across an available area specifically includes: determining the number N of coded slices and the number P of copies according to the number N of available areas and the number f of abnormal slices; for example, a correspondence table between the number N of available areas, the number f of abnormal slices, the number N of coded slices, and the number P of copies may be preset, where the correspondence maintained in the correspondence table may enable abnormality to occur in M (e.g., m=1) available areas, and if there are f abnormal slices, the read-write operation may still be normally performed; the number N of the coded slices and the number P of the copies can be determined from the corresponding relation table according to the number N of the available areas and the number f of the abnormal slices; the number of data slices m is preset, for example, m may be 12; in order to ensure that the read-write flow across the available areas is as small as possible, the data redundancy is relatively low and the disaster recovery service requirement is met, according to the number P of fragments to be stored, m+n, K persistent storage nodes are selected from at least P available areas, P is less than or equal to K (P is less than or equal to m+n), for example, one persistent storage node can be selected for each fragment, or one persistent storage node can be selected for two fragments, and the method is not limited to the method; and (3) locating different copies of the m data fragments in different available areas, ensuring that each available area in the P available areas contains one copy, reading data locally, and establishing a storage mapping relation between the P x m+n fragments and K persistence storage nodes at least according to the principle that different copies of the m data fragments are located in different available areas.
A data storage layout information table in the case of the different available area number N and the abnormal piece number f is provided below, as in table 1 of the following pages. Wherein d1-dm is a data slice, p1-p6 bit codes slice, AZ1-AZ5 is a usable area. Table 1 illustrates that the number of copies of m data slices is 2. Further, table 1 also includes: any number of shard damages that can be tolerated, the write flow amplification across the usable area, and the read flow amplification across the usable area and the data redundancy ratio after any 1 usable area or 1 shard anomaly. The read traffic amplification factor across the usable area refers to the ratio of the read traffic across the usable area to the user read data, and the data redundancy ratio refers to the ratio of the user data to the stored data.
Optionally, the implementation of selecting K persistent storage nodes from at least P available areas according to the number of tiles p×m+n to be stored is not limited. The following is an example.
For example, K persistent storage nodes are selected from among the P available areas according to the number of tiles p×m+n to be stored. For example, in table 1, in the case of p=2, n=2, f=1, 2,3, K persistent storage nodes are selected from the 2 available areas AZ1 and AZ 2. Accordingly, an embodiment for establishing a storage mapping relationship between p×m+n slices and K persistent storage nodes at least according to the principle that different copies of m data slices are located in different available areas includes: and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P available areas in a scattered way.
For another example, K persistent storage nodes are selected from among the p+1 available areas according to the number of tiles p×m+n to be stored. For example, in table 1, in the case of p=2, n=3, f=1, 2,3, K persistent storage nodes are selected from the 3 available areas AZ1, AZ2, and AZ 3; in the case of p=2, n=4, 5, and f=1, K persistent storage nodes are selected from the 3 available areas AZ1, AZ2, and AZ 3. Correspondingly, at least according to the principle that different copies of m data fragments are located in different available areas, the storage mapping relation between the p+n fragments and the K persistent storage nodes is established as follows: and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P+1 available areas in a scattered manner.
Optionally, when p=n-1, in the case of selecting one persistent storage node for each data slice and code slice, the number of persistent storage nodes k= (N-1) ×m+n. Further alternatively, n=3, k=2×m+n. For example, in the case where n=3, f=2, where AZ1 includes data slices d1-dm and code slices p1, AZ2 includes data slices d1-dm and code slices p2, AZ3 includes code slices p3, and one persistent storage node is selected for each data slice and code slice, the number of K is 2×m+n.
Table 1 is as follows:
for another example, K persistent storage nodes are selected from among the p+2 available areas according to the number of tiles p×m+n to be stored. For example, in table 1, in the case of p=2, n=3, f=2, 3, and n=5, f=2, K persistent storage nodes are selected from the 4 available areas AZ1, AZ2, AZ3, and AZ 4. Correspondingly, at least according to the principle that different copies of m data fragments are located in different available areas, the storage mapping relation between the p+n fragments and the K persistent storage nodes is established as follows: and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P+2 available areas in a scattered manner.
For another example, K persistent storage nodes are selected from within p+3 available regions according to the number of tiles p×m+n to be stored. For example, in table 1, in the case of p=2, n=5, f=3, K persistent storage nodes are selected from AZ1, AZ2, AZ3, AZ4, and AZ5 and the 5 available areas. Correspondingly, at least according to the principle that different copies of m data fragments are located in different available areas, the storage mapping relation between the p+n fragments and the K persistent storage nodes is established as follows: and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P+3 available areas in a scattered manner.
In an alternative embodiment, the computing node 105 may implement read and write operations on the cloud storage disk through the cloud storage access node 106. The storage service node 102 may respond to a read-write request initiated by the computing node to the cloud storage disk, and perform data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information. The procedure of the write operation and the read operation are described below, respectively.
Case 1: and (3) writing operation. The computing node 105 initiates a write request to the cloud storage disk, wherein the write request comprises data to be stored; the storage service node 102 responds to a write request initiated by a computing node to a cloud storage disk, and obtains data to be stored; the method comprises the steps of performing fragmentation and erasure code coding processing on data to be stored by adopting the number m of data fragments and the number n of coding fragments in erasure code to obtain m data fragments and n coding fragments; and storing the P.m data fragments and the n coding fragments into the K persistence storage nodes according to a storage mapping relation between the P.m+n fragments and the K persistence storage nodes included in the data storage layout information.
Optionally, an embodiment for storing p×m data slices and n code slices into K persistent storage nodes specifically according to a storage mapping relationship between p×m+n slices and K persistent storage nodes included in the data storage layout information includes: storing the same copy of the m data fragments into persistent storage nodes located in the same available area, wherein the persistent storage nodes located in the same available area belong to K persistent storage nodes, and correspondingly, P different copies are stored into P different available areas; the n code fragments are stored in at least one of the K persistent storage nodes in a decentralized manner.
Further alternatively, n code slices may be stored in the P available areas in a dispersed manner, for example, in table 1, in the case of p=2, n=2, f=1, 2,3, n code slices are dispersed in the 2 available areas AZ1 and AZ 2; n coded slices may also be stored scattered in p+1 available areas, for example, in table 1, in the case of p=2, n=3, f=1, 2,3, n coded slices are scattered in 3 available areas AZ1, AZ2, and AZ 3; in the case of p=2, n=4, 5, and f=1, n coded slices are dispersed in the 3 available regions AZ1, AZ2, and AZ 3; it is also possible to store N coded slices in p+2 or p+3 available areas, for example, in table 1, in the case of p=2, n=3, f=2, 3, and n=5, f=2, N coded slices are dispersed in 4 available areas AZ1, AZ2, AZ3, and AZ 4; also for example, in table 1, in the case of p=2, n=5, f=3, n coding slices are dispersed in AZ1, AZ2, AZ3, AZ4, and AZ5 and the 5 available regions.
In either case, the storage service node provides the corresponding data slice or code slice to the corresponding persistent storage node, which may store the corresponding data slice and code slice. Optionally, the K persistent storage nodes return storage success information to the storage service node after successfully storing the corresponding data fragments or the code fragments.
In an alternative embodiment, K persistent storage nodes may provide multiple persistent files through virtualization techniques, each persistent file corresponding to a Stripe (Stripe) group, each Stripe group including multiple stripes corresponding to physical address information (e.g., 4K or 8K, etc.) on K persistent storage nodes, e.g., each Stripe corresponding to a physical storage unit on one persistent storage node. Each stripe group is used to store P x m + n slices, e.g., one slice per stripe. That is, the file address information of each persistent file has a correspondence with the physical address information of p×m+n pieces on K persistent storage nodes.
Based on this, in the case that the K persistent storage nodes return storage success information to the storage service node after successfully storing the corresponding data fragments or the code fragments, the storage service node 102 receives the storage success information returned by the K persistent storage nodes after successfully storing the data fragments or the code fragments, respectively; and recording the mapping relation between the first LBA address information of the data to be stored in the cloud storage disk and the first file address information according to the storage success message, wherein the first file address information corresponds to the physical address information of the fragments corresponding to the data to be stored on the K persistent storage nodes. For example, the first file address information may be < persistence layer file name, offset address, data length >, the offset address and data length, and the length of the fragment, and the fragment corresponding to the first file address information may be determined.
In the case of storing the data to be stored in K persistent storage nodes, the data length of the data to be stored may be accumulated to a certain length threshold, and then split and encoded to obtain p×m+n slices, and stored in K persistent storage nodes. The length of data to be stored in each write request may not be sufficient for slicing and encoding, for example, the data length in the write request is 4K, and slicing and encoding are performed when the data length of the data to be stored is 16K, so that the data to be stored in the write request may be accumulated, and after reaching a certain data length, the slicing and encoding are performed to obtain p+m+n slices, and the p+n slices are stored in K persistent storage nodes.
In an alternative embodiment, the persistence management node may also monitor the running state of the available areas to which the K persistence storage nodes belong; when the abnormal first available area is detected, storing the first data fragments required to be stored in the first available area into other available areas, and after the first available area is restored, migrating the first data fragments into a persistent storage node in the first available area; or, storing the first code fragments required to be stored in the first available area into other available areas, and after the first available area is restored, migrating the first code fragments into a persistent storage node in the first available area; or, storing the first data fragments and the first coding fragments which are required to be stored in the first available area into other available areas, and after the first available area is restored, migrating the first data fragments and the first coding fragments into a persistent storage node in the first available area.
Case 2: and (3) a read operation. The compute node 105 initiates a read request to the cloud storage disk. The storage service node 102 responds to a read request initiated by the computing node to the cloud storage disk, and acquires second LBA address information of data to be read in the cloud storage disk; determining second file address information corresponding to the second LBA address information according to the second LBA address information and the mapping relation between each LBA address information and the file address information in the pre-recorded cloud storage disk; and acquiring physical address information of at least one fragment corresponding to the data to be read on the K persistent storage nodes according to the second file address information and the data storage layout information. For example, the second file address information is < the file name of the persistence layer, the offset address and the data length >, and according to the offset address and the data length, the data fragment corresponding to the data to be read can be determined by combining the data length of the fragment; determining storage position information of the P+n fragments corresponding to the file names of the persistence layer on the K persistence storage nodes according to the storage mapping relation between the P+m fragments and the K persistence storage nodes in the data storage layout information; and determining physical address information of at least one data fragment corresponding to the data to be read on the K persistent storage nodes according to storage position information of the P+n fragments corresponding to the file names of the persistent layer on the K persistent storage nodes. And reading at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to the physical address information of the at least one fragment corresponding to the data to be read on the K persistent storage nodes.
In an alternative embodiment, according to the physical address information of at least one data fragment corresponding to the data to be read on the K persistent storage nodes, in the process of reading at least one data fragment corresponding to the data to be read from the corresponding persistent storage node, there may be an abnormal usable area or a situation of abnormal fragments, where the number of abnormal usable areas is less than or equal to M and the number of abnormal fragments in the non-abnormal usable area is less than or equal to f, the storage service node may still read the data normally, and an exemplary description is given below.
Example A1: if the local available area where the storage service node is located is normal, reading at least one data fragment from the persistent storage node in the local available area according to physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the local available area;
example A2: if the local available area is abnormal and at least one data fragment corresponding to the data to be read in the non-abnormal available area is normal, reading the at least one data fragment from the persistent storage node in the non-abnormal available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the non-abnormal available area;
Example A3: if the local available area is abnormal and the abnormal data fragments exist in at least one data fragment corresponding to the data to be read in the non-abnormal available area, reading m normal fragments from the non-abnormal available area, and recovering the abnormal data fragments according to the m normal fragments; and acquiring at least one data fragment corresponding to the data to be read from the m normal fragments and the recovered abnormal fragments.
In the following, a scenario embodiment is provided, in which a cloud storage disk spans 3 AZ as an example is described as data storage layout information meeting the aforementioned disaster recovery requirement (a-d) across the available area.
Specific parameters are as follows: the number of available areas n=3, the available areas are: AZ1, AZ2 and AZ3. Number of data fragments: m=12; the number of tolerable fragmentation anomalies f=2 while there is 1 AZ anomaly; the user service selects deployed available areas AZ1 (main) and AZ2 (standby); data storage layout information for data slice d and code slice p:
AZ1: d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, p1；
AZ2: d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, p2；
AZ3: p3。
the following describes how data storage layout information meets disaster recovery requirements (a-d) across the available areas:
1. under the conditions of 1 AZ abnormality and f=2 slicing abnormality, normal reading and writing can still be performed.
Assuming that AZ1 is abnormal, fragments dx and dy (x is not equal to y, and x, y is not equal to {1-12 }) of AZ2 are also abnormal at the same time, if the data of dz is read and dz is not equal to dx or dy, the corresponding fragments dz are directly read from AZ 2; if dz is equal to dx or dy, then dz is regenerated (12 normal slices are required) by erasure code reconstruction, where AZ2 includes 11 normal slices (AZ 2 has 13 slices total, two abnormal slices, normal slice 13-2=11), AZ3 has 1 normal slice, and therefore, slice p3 of AZ3 is obtained, and the data slice of dz is reconstructed based on the normal 11 slices and slice p 3.
2. For a new write request, when a certain AZ is abnormal, the persistence management node 103 constructs new data storage layout information, temporarily stores the write request for the abnormal AZ in other AZs, and after the abnormal AZ returns to normal, migrates the data fragments in the other AZs, in which the abnormal AZ needs to be written, to the AZ that returns to normal, so as to meet the disaster recovery requirement. For example, AZ1 is abnormal, the persistence management node 103 notifies the storage service node to store the data fragments d1 to d12 to be written into AZ1 in AZ2 and AZ3, respectively, p1, p2 in AZ2, and p3 in AZ3. After AZ1 is restored, d1-d12 in AZ3 is migrated back to AZ1, and p1 in AZ2 is migrated back to AZ1.
3. The data storage layout information enables the user to read the request at AZ1/AZ2 to be completed locally. For a write request, the traffic across the available area is 13 slices (12 data slices and 1 code slice) in AZ2 and 1 code slice in AZ3, the user data amount is 12 data slices, and the traffic across the available area is (13+1)/12=1.17 times (minimum 1) the user data amount. While 3 copies of data store layout information (3 available areas each store one copy), traffic needs to be written to the other 2 available areas, i.e. the traffic across the available areas is 2 times the amount of user data. Compared with a 3-copy data layout strategy, the data storage layout information provided by the embodiment of the application has better performance in the cross-available-area write flow.
4. When an abnormality occurs in a certain data slice of AZ1, a corresponding data read request is sent to AZ2, and therefore, the data traffic magnification across the available area is 1. Whereas in the data encoding and layout scheme of EC (12, 9), 3 available regions, 12 data slices per copy, 9 coding slices, each AZ comprising 4 data slices and 3 coding slices; for example, one data slice in AZ1 is abnormal, 3 data and 3 code total 6 normal slices remain, and to construct the abnormal data slice, 3 code slices need to be acquired from AZ2 and AZ3 respectively, and the abnormal slice is constructed by reading 6 normal slices across the available area. Thus, the scheme of EC (12, 9) has a data read amplification across the available area of 6. Compared with the scheme of EC (12, 9) layout, the data storage layout information provided by the embodiment of the application has better data reading performance across the available areas.
5. The data storage layout information provided by the embodiment of the application enables the data redundancy ratio to be (12+12+3)/12=2.25, and is reduced by (3-2.25)/3=25% compared with the 3-copy data redundancy ratio (3/1).
In the embodiment of the application, the logic storage cluster deployed across the available areas has strong expandability (the number N of the available areas is supported to be variable), internal data synchronization and disaster recovery details are shielded for users, the cloud storage disks of the logic across N available areas are presented for the users, and the users do not need to understand and maintain the binding relationship between the copy logic between the master available areas and the slave cloud storage disks. In addition, the data cross-availability area high availability is realized directly through the data consistency read-write mechanism of the bottom distributed storage, a master-slave replication mechanism is not needed, and the overall complexity of the cloud storage system is reduced. Moreover, by adopting a near scheduling access mode and adopting a multi-copy erasure code fusion redundancy coding and layout mode for data reading and writing, the method not only meets the requirements of better performance and elasticity of reading and writing, but also can reduce the requirement of network bandwidth crossing an available area, and keeps a lower data redundancy ratio. In addition to providing the system embodiment, the embodiment of the application also provides a data reading and writing method, and the process of the data reading and writing method provided by the embodiment of the application is explained below.
Fig. 2a is a flow chart of a data read-write method according to an exemplary embodiment of the present application, where the method is applicable to a storage service management node in a cloud storage system, and as shown in fig. 2a, the method includes:
201a, creating a cloud storage disk which spans N available areas for a computing node, wherein N is more than or equal to 2, and N is a positive integer; the cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation;
201b, distributing storage service nodes for the cloud storage disk, wherein the distributed storage service nodes perform data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information, and the data storage layout information is generated by the persistent management nodes in the cloud storage system according to the disaster recovery requirement information of the cross-available area of the cloud storage disk.
In an alternative embodiment, the storage service nodes are multiple, and the storage service nodes are distributed in different available areas; accordingly, allocating a storage service node for the cloud storage disk includes: according to the available areas to which the storage service nodes belong, a nearby scheduling strategy is adopted to allocate the storage service nodes for the cloud storage disk, so that the storage service nodes responding to the read-write request of the cloud storage disk and the cloud storage access node initiating the read-write request are ensured to be in the same available area as much as possible, and the response speed is improved.
Fig. 2b is a flowchart of another data read-write method according to an exemplary embodiment of the present application, where the method is applicable to a persistent management node in a cloud storage system, and as shown in fig. 2b, the method includes:
201b, acquiring disaster recovery requirement information of a cloud storage disk in a cross-available area; the cloud storage disk spans N available areas and adopts a redundancy storage mode of fusion of multiple copies and erasure codes, wherein the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer;
202b, generating data storage layout information required for reading and writing the cloud storage disk according to the disaster recovery requirement information of the cross-available area, wherein the data storage layout information is used for describing the storage mode of the coded fragments and the multi-copy data fragments in the persistent storage nodes of the cross-available area;
203b, providing the data storage layout information to the storage service nodes, so that the distributed storage service nodes can perform data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information.
In an alternative embodiment, the disaster recovery requirement information across the available areas includes the number f of abnormal slices that can be tolerated when M available areas in the N available areas are abnormal, wherein M is greater than or equal to 1 and less than N, f is greater than or equal to 1, and M and f are both positive integers; the data storage layout information specifically comprises storage mapping relations between P+n fragments and K persistent storage nodes, wherein the K persistent storage nodes are at least distributed in P available areas; wherein m and N are the number of data slices and code slices in the erasure code, P represents the number of copies of m data slices, different copies of m data slices are located in different available areas, P is not less than 2 and not more than (N-1), N is not less than f, P, K, m and N are positive integers.
In an alternative embodiment, generating data storage layout information required for reading and writing the cloud storage disk according to disaster tolerance requirement information across the available area includes: determining the number N of coded fragments and the number P of copies according to the number N of available areas and the number f of abnormal fragments, wherein the number m of data fragments is preset; selecting K persistent storage nodes from at least P available areas according to the number P+m of fragments to be stored, wherein P is less than or equal to K (P+m+n); and establishing a storage mapping relation between the P.m+n fragments and the K persistence storage nodes at least according to the principle that different copies of the m data fragments are positioned in different available areas.
Optionally, selecting K persistent storage nodes from at least P available areas according to the number of tiles p×m+n to be stored, including: selecting K persistent storage nodes from P+1 available areas according to the number P of fragments to be stored; accordingly, establishing a storage mapping relationship between p+n pieces and K persistent storage nodes at least according to the principle that different copies of m data pieces are located in different available areas, including: and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P+1 available areas in a scattered manner.
Further alternatively, p=n-1, and k= (N-1) ×m+n.
Further alternatively, n=3, k=2×m+n.
Fig. 2c is a flow chart of another data read-write method according to an exemplary embodiment of the present application, where the method is applicable to a storage service node in a cloud storage system, as shown in fig. 2c, and the method includes:
201c, receiving data storage layout information required by reading and writing a cloud storage disk and transmitted by a persistence management node in a cloud storage system, wherein the data storage layout information is generated by the persistence management node according to disaster tolerance requirement information of the cloud storage disk across an available area;
202c, responding to a read-write request initiated by a computing node to a cloud storage disk, and performing data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information; the cloud storage disk spans N available areas, a redundancy storage mode of fusion of multiple copies and erasure codes is adopted, the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer.
In an alternative embodiment, receiving data storage layout information required for reading and writing a cloud storage disk sent by a persistence management node in a cloud storage system includes: when a write request initiated by a computing node to a cloud storage disk is received for the first time, data storage layout information required for reading and writing the cloud storage disk is acquired from a persistence management node in a cloud storage system.
In an alternative embodiment, the disaster recovery requirement information across the available areas includes the number f of abnormal slices that can be tolerated when M available areas in the N available areas are abnormal, wherein M is greater than or equal to 1 and less than N, f is greater than or equal to 1, and M and f are both positive integers; the data storage layout information specifically comprises storage mapping relations between P+m pieces and K persistent storage nodes, wherein the K persistent storage nodes are at least distributed in P available areas; wherein m and N are the number of data slices and code slices in the erasure code respectively, P represents the number of copies of m data slices, different copies of m data slices are located in different available areas, P is more than or equal to 2 and less than or equal to (N-1), N is more than or equal to f, P, K, m and N are positive integers; correspondingly, responding to a read-write request initiated by a computing node to a cloud storage disk, and performing data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information, wherein the data read-write operation comprises the following steps: responding to a write request initiated by a computing node to a cloud storage disk, and acquiring data to be stored; the method comprises the steps of performing fragmentation and erasure code coding processing on data to be stored by adopting the number m of data fragments and the number n of coding fragments in erasure code to obtain m data fragments and n coding fragments; and storing the P.m data fragments and the n coding fragments into the K persistence storage nodes according to a storage mapping relation between the P.m+n fragments and the K persistence storage nodes included in the data storage layout information.
Optionally, storing the p×m data slices and the n code slices in the K persistent storage nodes according to a storage mapping relationship between the p×m+n slices and the K persistent storage nodes included in the data storage layout information includes: storing the same copy of the m data fragments into the persistent storage nodes located in the same available area in the K persistent storage nodes, and storing P different copies into the persistent storage nodes in P different available areas; the n code fragments are stored in at least one of the K persistent storage nodes in a decentralized manner.
Further alternatively, the K persistent storage nodes are distributed in the p+1 available areas, and then the n code fragments are stored in at least one of the K persistent storage nodes in a scattered manner, including: the n code slices are stored in the P+1 available areas in a scattered manner.
In an alternative embodiment, the method provided in the embodiment of the present application further includes: receiving storage success information returned by the K persistence storage nodes after the data fragments or the code fragments are successfully stored respectively; and recording the mapping relation between the first LBA address information of the data to be stored in the cloud storage disk and the first file address information according to the storage success message, wherein the first file address information corresponds to the physical address information of the fragments corresponding to the data to be stored on the K persistent storage nodes.
In an alternative embodiment, the method provided in the embodiment of the present application further includes: monitoring the running states of available areas of the K persistent storage nodes; and when the abnormal first available area is detected, storing the first data fragments and/or the first coding fragments which are required to be stored in the first available area into other available areas, and after the first available area is restored, migrating the first data fragments and/or the first coding fragments into a persistent storage node in the first available area.
In an alternative embodiment, responding to a read-write request initiated by a computing node to a cloud storage disk, and performing data read-write operation on a persistent storage node in a cross-available area according to data storage layout information, including: responding to a read request initiated by a computing node to a cloud storage disk, and acquiring second LBA address information of data to be read in the cloud storage disk; determining second file address information according to the second LBA address information and the mapping relation between each LBA address information and file address information in the pre-recorded cloud storage disk; according to the second file address information and the data storage layout information, acquiring physical address information of at least one data fragment corresponding to the data to be read on the K persistence storage nodes; and reading at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to the physical address information of the at least one data fragment corresponding to the data to be read on the K persistent storage nodes.
Optionally, according to physical address information of at least one data fragment corresponding to the data to be read on the K persistent storage nodes, reading the at least one data fragment corresponding to the data to be read from the corresponding persistent storage node includes: if the number of the abnormal available areas is less than or equal to M and the number of the abnormal fragments in the non-abnormal available areas is less than or equal to f, if the local available area where the storage service node is located is normal, reading at least one data fragment from the persistent storage node in the local available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the local available area; if the local available area is abnormal and at least one data fragment corresponding to the data to be read in the non-abnormal available area is normal, reading the at least one data fragment from the persistent storage node in the non-abnormal available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the non-abnormal available area; if the local available area is abnormal and the abnormal data fragments exist in at least one data fragment corresponding to the data to be read in the non-abnormal available area, reading m normal fragments from the non-abnormal available area, and recovering the abnormal data fragments according to the m normal fragments; and acquiring at least one data fragment corresponding to the data to be read from the m normal fragments and the recovered abnormal fragments.
The detailed implementation and the advantages of the steps in the methods shown in fig. 2 a-2 c provided in the embodiments of the present application have been described in detail in the foregoing embodiments, and will not be described in detail herein.
It should be noted that, the execution subjects of each step of the method provided in the above embodiment may be the same device, or the method may also be executed by different devices. For example, the execution subject of step 201a to step 203a may be device a; for another example, the execution subject of steps 201a and 202a may be device a, and the execution subject of step 203a may be device B; etc.
In addition, in some of the flows described in the above embodiments and the drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that the operations may be performed out of the order in which they appear herein or performed in parallel, the sequence numbers of the operations such as 201a, 202a, etc. are merely used to distinguish between the various operations, and the sequence numbers themselves do not represent any order of execution. In addition, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first" and "second" herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, and are not limited to the "first" and the "second" being different types.
Fig. 3a is a schematic structural diagram of a data read-write device according to an exemplary embodiment of the present application, where the device corresponds to a storage service management node in a cloud storage system, and as shown in fig. 3a, the device includes: a creation module 31a and an allocation module 32a.
The creation module 31a is configured to create a cloud storage disk spanning N available areas for a computing node, where N is greater than or equal to 2, and N is a positive integer; the cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation;
the allocation module 32a is configured to allocate storage service nodes to the cloud storage disk, so that the allocated storage service nodes perform data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information, where the data storage layout information is generated by the persistent management nodes in the cloud storage system according to the disaster tolerance requirement information of the cross-available area of the cloud storage disk.
In an alternative embodiment, the storage service nodes are multiple, and the storage service nodes are distributed in different available areas; the distribution module is specifically used for: according to the available areas to which the storage service nodes belong, a nearby scheduling strategy is adopted to allocate the storage service nodes for the cloud storage disk, so that the storage service nodes responding to the read-write request of the cloud storage disk and the cloud storage access node initiating the read-write request are ensured to be in the same available area as much as possible, and the response speed is improved.
Fig. 3b is a schematic structural diagram of another data read-write device according to an exemplary embodiment of the present application, corresponding to a persistence management node in a cloud storage system, where the device includes: an acquisition module 31b, a generation module 32b and a provision module 33b.
An obtaining module 31b, configured to obtain disaster recovery requirement information of a cloud storage disk in a cross-available area; the cloud storage disk spans N available areas and adopts a redundancy storage mode of fusion of multiple copies and erasure codes, wherein the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer;
the generating module 32b is configured to generate data storage layout information required for reading and writing to the cloud storage disk according to the disaster tolerance requirement information of the cross-available area, where the data storage layout information is used to describe a storage mode of the encoded fragments and the multi-copy data fragments in the persistent storage nodes of the cross-available area;
and the providing module 33b is configured to provide the data storage layout information to the storage service node, so that the distributed storage service node performs data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information.
In an alternative embodiment, the disaster recovery requirement information across the available areas includes the number f of abnormal slices that can be tolerated when M available areas in the N available areas are abnormal, wherein M is greater than or equal to 1 and less than N, f is greater than or equal to 1, and M and f are both positive integers; the data storage layout information specifically comprises storage mapping relations between P+n fragments and K persistent storage nodes, wherein the K persistent storage nodes are at least distributed in P available areas; wherein m and N are the number of data slices and code slices in the erasure code, P represents the number of copies of m data slices, different copies of m data slices are located in different available areas, P is not less than 2 and not more than (N-1), N is not less than f, P, K, m and N are positive integers.
In an alternative embodiment, the generating module is specifically configured to: determining the number N of coded slices and the number P of copies according to the number N of available areas and the number f of abnormal slices, wherein the number m of data slices is preset; selecting K persistent storage nodes from at least P available areas according to the number P+m of fragments to be stored, wherein P is less than or equal to K (P+m+n); and establishing a storage mapping relation between the P.m+n fragments and the K persistence storage nodes at least according to the principle that different copies of the m data fragments are positioned in different available areas.
Optionally, the generating module is specifically configured to: selecting K persistent storage nodes from P+1 available areas according to the number P of fragments to be stored; and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P+1 available areas in a scattered manner.
Further alternatively, p=n-1, and k= (N-1) ×m+n.
Further alternatively, n=3, k=2×m+n.
Fig. 3c is a schematic structural diagram of still another data read-write device according to an exemplary embodiment of the present application, where the device corresponds to a storage service node in a cloud storage system, and the device includes: a receiving module 31c and a reading and writing module 32c.
The receiving module 31c is configured to receive data storage layout information required for reading and writing a cloud storage disk, which is sent by a persistence management node in the cloud storage system, where the data storage layout information is generated by the persistence management node according to disaster tolerance requirement information of the cloud storage disk across an available area;
the read-write module 32c is configured to respond to a read-write request initiated by the computing node to the cloud storage disk, and perform a data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information; the cloud storage disk spans N available areas, a redundancy storage mode of fusion of multiple copies and erasure codes is adopted, the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer.
In an alternative embodiment, the receiving module is specifically configured to: when a write request initiated by a computing node to a cloud storage disk is received for the first time, data storage layout information required for reading and writing the cloud storage disk is acquired from a persistence management node in a cloud storage system.
In an alternative embodiment, the disaster recovery requirement information across the available areas includes the number f of abnormal slices that can be tolerated when M available areas in the N available areas are abnormal, wherein M is greater than or equal to 1 and less than N, f is greater than or equal to 1, and M and f are both positive integers; the data storage layout information specifically comprises storage mapping relations between P+m pieces and K persistent storage nodes, wherein the K persistent storage nodes are at least distributed in P available areas; wherein m and N are the number of data slices and code slices in the erasure code respectively, P represents the number of copies of m data slices, different copies of m data slices are located in different available areas, P is more than or equal to 2 and less than or equal to (N-1), N is more than or equal to f, P, K, m and N are positive integers; the read-write module is specifically used for: responding to a write request initiated by a computing node to a cloud storage disk, and acquiring data to be stored; the method comprises the steps of performing fragmentation and erasure code coding processing on data to be stored by adopting the number m of data fragments and the number n of coding fragments in erasure code to obtain m data fragments and n coding fragments; and storing the P.m data fragments and the n coding fragments into the K persistence storage nodes according to a storage mapping relation between the P.m+n fragments and the K persistence storage nodes included in the data storage layout information.
Optionally, the read-write module is specifically configured to: storing the same copy of the m data fragments into the persistent storage nodes located in the same available area in the K persistent storage nodes, and storing P different copies into the persistent storage nodes in P different available areas; the n code fragments are stored in at least one of the K persistent storage nodes in a decentralized manner.
Further alternatively, the K persistent storage nodes are distributed in p+1 available areas, and the read-write module is specifically configured to: the n code slices are stored in the P+1 available areas in a scattered manner.
In an optional embodiment, the data read-write device provided in the embodiment of the present application further includes: a recording module; the receiving module is used for receiving storage success information returned by the K persistence storage nodes after successful data fragmentation or code fragmentation are stored respectively; and the recording module is used for recording the mapping relation between the first LBA address information of the data to be stored in the cloud storage disk and the first file address information according to the storage success message, wherein the first file address information corresponds to the physical address information of the fragments corresponding to the data to be stored on the K persistence storage nodes.
In an optional embodiment, the data read-write device provided in the embodiment of the present application further includes: the device comprises a monitoring module, a storage module and a migration module; the monitoring module is used for monitoring the running states of the available areas of the K persistent storage nodes; the storage module is used for storing the first data fragments and/or the first code fragments which are required to be stored in the first available area into other available areas when the first available area with the abnormality is detected, and the migration module is used for migrating the first data fragments and/or the first code fragments into a persistent storage node in the first available area after the first available area is restored.
In an alternative embodiment, the read-write module is specifically configured to: responding to a read request initiated by a computing node to a cloud storage disk, and acquiring second LBA address information of data to be read in the cloud storage disk; determining second file address information according to the second LBA address information and the mapping relation between each LBA address information and file address information in the pre-recorded cloud storage disk; according to the second file address information and the data storage layout information, acquiring physical address information of at least one data fragment corresponding to the data to be read on the K persistence storage nodes; and reading at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to the physical address information of the at least one data fragment corresponding to the data to be read on the K persistent storage nodes.
Optionally, the read-write module is specifically configured to: if the number of the abnormal available areas is less than or equal to M and the number of the abnormal fragments in the non-abnormal available areas is less than or equal to f, if the local available area where the storage service node is located is normal, reading at least one data fragment from the persistent storage node in the local available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the local available area; if the local available area is abnormal and at least one data fragment corresponding to the data to be read in the non-abnormal available area is normal, reading the at least one data fragment from the persistent storage node in the non-abnormal available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the non-abnormal available area; if the local available area is abnormal and the abnormal data fragments exist in at least one data fragment corresponding to the data to be read in the non-abnormal available area, reading m normal fragments from the non-abnormal available area, and recovering the abnormal data fragments according to the m normal fragments; and acquiring at least one data fragment corresponding to the data to be read from the m normal fragments and the recovered abnormal fragments.
The detailed implementation and advantages of the steps in the apparatus shown in fig. 3 a-3 c provided in the embodiments of the present application have been described in detail in the previous embodiments, and will not be described in detail herein.
Fig. 4 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application, where the electronic device corresponds to a storage service management node in a cloud storage system, and as shown in fig. 4, the device includes: a memory 44 and a processor 45.
Memory 44 is used to store computer programs and may be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on an electronic device, and the like.
A processor 45 coupled to the memory 44 for executing the computer program in the memory 44 for: creating a cloud storage disk which spans N available areas for a computing node, wherein N is more than or equal to 2, and N is a positive integer; the cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation; and distributing storage service nodes for the cloud storage disk, wherein the distributed storage service nodes perform data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information, and the data storage layout information is generated by the persistent management nodes in the cloud storage system according to the disaster recovery requirement information of the cross-available area of the cloud storage disk.
In an alternative embodiment, the storage service nodes are multiple, and the storage service nodes are distributed in different available areas; the processor 45 is specifically configured to, when allocating a storage service node to a cloud storage disk: according to the available areas to which the storage service nodes belong, a nearby scheduling strategy is adopted to allocate the storage service nodes for the cloud storage disk, so that the storage service nodes responding to the read-write request of the cloud storage disk and the cloud storage access node initiating the read-write request are ensured to be in the same available area as much as possible, and the response speed is improved.
Further, as shown in fig. 4, the electronic device further includes: communication component 46, display 47, power supply component 48, audio component 49, and other components. Only some of the components are schematically shown in fig. 4, which does not mean that the electronic device only comprises the components shown in fig. 4. In addition, the components within the dashed box in fig. 4 are optional components, not necessarily optional components, depending on the product form of the electronic device. The electronic device of the embodiment can realize the server-side devices such as a conventional server, a cloud server or a server array.
The embodiment of the application also provides an electronic device, corresponding to the persistent management node in the cloud storage system, wherein the implementation structure of the electronic device is the same as or similar to that of the electronic device shown in fig. 4, and the implementation can be realized by referring to the structure of the electronic device shown in fig. 4. The electronic device provided in this embodiment is different from the electronic device in the embodiment shown in fig. 4 mainly in that: the functions implemented by a processor executing a computer program stored in memory are different. The electronic device provided in this embodiment may have a processor executing a computer program stored in a memory, and may be configured to: acquiring disaster recovery requirement information of a cloud storage disk in a cross-available area; the cloud storage disk spans N available areas and adopts a redundancy storage mode of fusion of multiple copies and erasure codes, wherein the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer; generating data storage layout information required for reading and writing the cloud storage disk according to the disaster recovery requirement information of the cross-available area, wherein the data storage layout information is used for describing the storage mode of the coded fragments and the multi-copy data fragments in the persistent storage nodes of the cross-available area; and providing the data storage layout information to the storage service nodes so that the distributed storage service nodes can perform data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information.
In an alternative embodiment, the disaster recovery requirement information across the available areas includes the number f of abnormal slices that can be tolerated when M available areas in the N available areas are abnormal, wherein M is greater than or equal to 1 and less than N, f is greater than or equal to 1, and M and f are both positive integers; the data storage layout information specifically comprises storage mapping relations between P+n fragments and K persistent storage nodes, wherein the K persistent storage nodes are at least distributed in P available areas; wherein m and N are the number of data slices and code slices in the erasure code, P represents the number of copies of m data slices, different copies of m data slices are located in different available areas, P is not less than 2 and not more than (N-1), N is not less than f, P, K, m and N are positive integers.
In an optional embodiment, when generating data storage layout information required for reading and writing to the cloud storage disk according to the disaster tolerance requirement information of the cross-available area, the processor is specifically configured to: determining the number N of coded slices and the number P of copies according to the number N of available areas and the number f of abnormal slices, wherein the number m of data slices is preset; selecting K persistent storage nodes from at least P available areas according to the number P+m of fragments to be stored, wherein P is less than or equal to K (P+m+n); and establishing a storage mapping relation between the P.m+n fragments and the K persistence storage nodes at least according to the principle that different copies of the m data fragments are positioned in different available areas.
Optionally, the processor is specifically configured to, when selecting K persistent storage nodes from at least P available areas according to the number of slices p×m+n to be stored: selecting K persistent storage nodes from P+1 available areas according to the number P of fragments to be stored; correspondingly, when the processor establishes a storage mapping relationship between p×m+n slices and K persistent storage nodes according to at least the principle that different copies of m data slices are located in different available areas, the processor is specifically configured to: and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P+1 available areas in a scattered manner.
Further alternatively, p=n-1, and k= (N-1) ×m+n.
Further alternatively, n=3, k=2×m+n.
The embodiment of the application also provides an electronic device, corresponding to the storage service node in the cloud storage system, where the implementation structure of the electronic device is the same as or similar to the implementation structure of the electronic device shown in fig. 4, and the implementation can be realized by referring to the structure of the electronic device shown in fig. 4. The electronic device provided in this embodiment is different from the electronic device in the embodiment shown in fig. 4 mainly in that: the functions implemented by a processor executing a computer program stored in memory are different. The electronic device provided in this embodiment may have a processor executing a computer program stored in a memory, and may be configured to: receiving data storage layout information required by reading and writing a cloud storage disk and transmitted by a persistence management node in a cloud storage system, wherein the data storage layout information is generated by the persistence management node according to disaster tolerance requirement information of the cloud storage disk in a cross-available area; responding to a read-write request initiated by a computing node to a cloud storage disk, and performing data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information; the cloud storage disk spans N available areas, a redundancy storage mode of fusion of multiple copies and erasure codes is adopted, the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer.
In an optional embodiment, when receiving data storage layout information required for reading and writing a cloud storage disk sent by a persistence management node in a cloud storage system, the processor is specifically configured to: when a write request initiated by a computing node to a cloud storage disk is received for the first time, data storage layout information required for reading and writing the cloud storage disk is acquired from a persistence management node in a cloud storage system.
In an alternative embodiment, the disaster recovery requirement information across the available areas includes the number f of abnormal slices that can be tolerated when M available areas in the N available areas are abnormal, wherein M is greater than or equal to 1 and less than N, f is greater than or equal to 1, and M and f are both positive integers; the data storage layout information specifically comprises storage mapping relations between P+m pieces and K persistent storage nodes, wherein the K persistent storage nodes are at least distributed in P available areas; wherein m and N are the number of data slices and code slices in the erasure code respectively, P represents the number of copies of m data slices, different copies of m data slices are located in different available areas, P is more than or equal to 2 and less than or equal to (N-1), N is more than or equal to f, P, K, m and N are positive integers; the processor is specifically configured to, when responding to a read-write request initiated by a computing node to a cloud storage disk and performing data read-write operation on a persistent storage node in a cross-available area according to data storage layout information: responding to a write request initiated by a computing node to a cloud storage disk, and acquiring data to be stored; the method comprises the steps of performing fragmentation and erasure code coding processing on data to be stored by adopting the number m of data fragments and the number n of coding fragments in erasure code to obtain m data fragments and n coding fragments; and storing the P.m data fragments and the n coding fragments into the K persistence storage nodes according to a storage mapping relation between the P.m+n fragments and the K persistence storage nodes included in the data storage layout information.
Optionally, when the processor stores the p×m data slices and the n code slices into the K persistent storage nodes according to a storage mapping relationship between the p×m+n slices and the K persistent storage nodes included in the data storage layout information, the processor is specifically configured to: storing the same copy of the m data fragments into the persistent storage nodes located in the same available area in the K persistent storage nodes, and storing P different copies into the persistent storage nodes in P different available areas; the n code fragments are stored in at least one of the K persistent storage nodes in a decentralized manner.
Further optionally, the K persistent storage nodes are distributed in the p+1 available areas, and the processor is specifically configured to, when the n code fragments are stored in at least one of the K persistent storage nodes in a scattered manner: the n code slices are stored in the P+1 available areas in a scattered manner.
In an alternative embodiment, the processor is further configured to: receiving storage success information returned by the K persistence storage nodes after the data fragments or the code fragments are successfully stored respectively; and recording the mapping relation between the first LBA address information of the data to be stored in the cloud storage disk and the first file address information according to the storage success message, wherein the first file address information corresponds to the physical address information of the fragments corresponding to the data to be stored on the K persistent storage nodes.
In an alternative embodiment, the processor is further configured to: monitoring the running states of available areas of the K persistent storage nodes; and when the abnormal first available area is detected, storing the first data fragments and/or the first coding fragments which are required to be stored in the first available area into other available areas, and after the first available area is restored, migrating the first data fragments and/or the first coding fragments into a persistent storage node in the first available area.
In an optional embodiment, when responding to a read-write request initiated by a computing node to a cloud storage disk, the processor is specifically configured to, when performing a data read-write operation on a persistent storage node across an available area according to data storage layout information: responding to a read request initiated by a computing node to a cloud storage disk, and acquiring second LBA address information of data to be read in the cloud storage disk; determining second file address information according to the second LBA address information and the mapping relation between each LBA address information and file address information in the pre-recorded cloud storage disk; according to the second file address information and the data storage layout information, acquiring physical address information of at least one data fragment corresponding to the data to be read on the K persistence storage nodes; and reading at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to the physical address information of the at least one data fragment corresponding to the data to be read on the K persistent storage nodes.
In an optional embodiment, when the processor reads at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to the physical address information of the at least one data fragment corresponding to the data to be read on the K persistent storage nodes, the processor is specifically configured to: if the number of the abnormal available areas is less than or equal to M and the number of the abnormal fragments in the non-abnormal available areas is less than or equal to f, if the local available area where the storage service node is located is normal, reading at least one data fragment from the persistent storage node in the local available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the local available area; if the local available area is abnormal and at least one data fragment corresponding to the data to be read in the non-abnormal available area is normal, reading the at least one data fragment from the persistent storage node in the non-abnormal available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the non-abnormal available area; if the local available area is abnormal and the abnormal data fragments exist in at least one data fragment corresponding to the data to be read in the non-abnormal available area, reading m normal fragments from the non-abnormal available area, and recovering the abnormal data fragments according to the m normal fragments; and acquiring at least one data fragment corresponding to the data to be read from the m normal fragments and the recovered abnormal fragments.
The detailed implementation and the beneficial effects of each step in the electronic device provided in the embodiments of the present application have been described in detail in the foregoing embodiments, and will not be described in detail herein.
Accordingly, the embodiments of the present application further provide a computer readable storage medium storing a computer program, where the computer program is capable of implementing the steps executable by the session state synchronization device in the method embodiments shown in fig. 2a to 2 c.
The Memory may be implemented by any type or combination of volatile or non-volatile Memory devices, such as Static Random-Access Memory (SRAM), electrically erasable programmable Read-Only Memory (Electrically Erasable Programmable Read Only Memory, EEPROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk.
The communication component is configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile communication network of WiFi,2G, 3G, 4G/LTE, 5G, etc., or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a near field communication (Near Field Communication, NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on radio frequency identification (Radio Frequency Identification, RFID) technology, infrared data association (Infrared Data Association, irDA) technology, ultra Wideband (UWB) technology, blueTooth (BT) technology, and other technologies.
The display includes a screen, which may include a liquid crystal display (Liquid Crystal Display, LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or sliding action, but also the duration and pressure associated with the touch or sliding operation.
The power supply component provides power for various components of equipment where the power supply component is located. The power components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the devices in which the power components are located.
The audio component described above may be configured to output and/or input an audio signal. For example, the audio component includes a Microphone (MIC) configured to receive external audio signals when the device in which the audio component is located is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in a memory or transmitted via a communication component. In some embodiments, the audio assembly further comprises a speaker for outputting audio signals.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, magnetic disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (Central Processing Unit, CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random access memory (Random Access Memory, RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase-change memory (Phase-change Random Access Memory, PRAM), static Random Access Memory (SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (Digital Video Disc, DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by the computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (14)
1. A cloud storage system, comprising: the system comprises a storage service management node, a storage service node, a persistence management node and a plurality of persistence storage nodes, wherein the persistence storage nodes are distributed in a plurality of available areas;
The storage service management node is used for creating a cloud storage disk crossing N available areas for a computing node, distributing storage service nodes for the cloud storage disk, enabling the distributed storage service nodes to respond to a read-write request initiated by the computing node to the cloud storage disk, wherein N is more than or equal to 2, and N is a positive integer; the cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation;
the persistence management node is used for acquiring disaster recovery requirement information of a cross-available area of the cloud storage disk, wherein the cross-available area disaster recovery requirement information comprises the number f of abnormal fragments which can be tolerated when M available areas in N available areas are abnormal; determining the number N of coded slices and the number P of copies according to the number N of the available areas and the number f of abnormal slices; selecting K persistent storage nodes from at least P available areas according to the number P of fragments to be stored; according to at least the principle that different copies of m data fragments are positioned in different available areas, establishing a storage mapping relation between P x m+n fragments and K persistence storage nodes as data storage layout information required for reading and writing the cloud storage disk; wherein M is more than or equal to 1 and less than or equal to N, f is more than or equal to 1, P is more than or equal to K is more than or equal to (P+n), P is more than or equal to 2 and less than or equal to (N-1), N is more than or equal to f, M and N are respectively the number of data fragments and coding fragments in the erasure code, the number M of the data fragments is preset, P represents the number of copies of the M data fragments, and M, f, P, K, M and N are both positive integers;
And the storage service node is used for responding to the read-write request initiated by the computing node to the cloud storage disk and performing data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information.
2. The system according to claim 1, wherein the storage service node is configured to, when responding to a read-write request initiated by the computing node to the cloud storage disk, perform a data read-write operation on a persistent storage node across the available area according to the data storage layout information:
responding to a write request initiated by the computing node to the cloud storage disk, and acquiring data to be stored;
performing slicing and erasure code coding processing on the data to be stored by adopting the number m of data slices and the number n of coding slices in erasure code to obtain m data slices and n coding slices;
and storing the P.m data fragments and the n coding fragments into the K persistence storage nodes according to a storage mapping relation between the P.m+n fragments and the K persistence storage nodes included in the data storage layout information.
3. The system according to any one of claims 1-2, wherein the storage service node is configured, when responding to a read-write request initiated by the computing node to the cloud storage disk, to perform a data read-write operation on a persistent storage node across an available area according to the data storage layout information, to:
Responding to a read request initiated by the computing node to the cloud storage disk, and acquiring second logic block addressing LBA address information of data to be read in the cloud storage disk;
determining second file address information according to the second LBA address information and a pre-recorded mapping relation between each LBA address information and file address information in the cloud storage disk;
according to the second file address information and the data storage layout information, physical address information of at least one data fragment corresponding to the data to be read on K persistence storage nodes is obtained;
and reading the at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to the physical address information of the at least one data fragment corresponding to the data to be read on the K persistent storage nodes.
4. A data read-write method, suitable for a persistent management node in a cloud storage system, the method comprising:
acquiring disaster recovery requirement information of a cloud storage disk in a cross-available area; the cloud storage disk spans N available areas and adopts a redundancy storage mode of fusion of multiple copies and erasure codes, the redundancy storage mode comprises coding fragmentation and multiple copies of data fragmentation, disaster recovery requirement information of the spans available areas comprises the number f of abnormal fragmentation which can be tolerated when M available areas in the N available areas are abnormal, wherein N is more than or equal to 2, M is more than or equal to 1 and less than N, f is more than or equal to 1, and N, M and f are both positive integers;
Determining the number N of coded slices and the number P of copies according to the number N of the available areas and the number f of abnormal slices; selecting K persistent storage nodes from at least P available areas according to the number P of fragments to be stored; according to at least the principle that different copies of m data fragments are positioned in different available areas, establishing a storage mapping relation between P x m+n fragments and K persistence storage nodes as data storage layout information required for reading and writing the cloud storage disk; wherein, P is less than or equal to K is less than or equal to (P+n), P is less than or equal to 2 is less than or equal to (N-1), N is more than or equal to f, m and N are respectively the number of data fragments and coding fragments in erasure codes, the number m of the data fragments is preset, P represents the number of copies of the m data fragments, and P, K, m and N are both positive integers;
and providing the data storage layout information to a storage service node so that the distributed storage service node can perform data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information.
5. The method of claim 4, wherein selecting K persistent storage nodes from at least P available regions according to the number of tiles P x m+n to be stored comprises:
Selecting K persistent storage nodes from P+1 available areas according to the number P of fragments to be stored;
accordingly, establishing a storage mapping relationship between p+n pieces and K persistent storage nodes at least according to the principle that different copies of m data pieces are located in different available areas, including:
and establishing a storage mapping relation between the P+n fragments and K persistence storage nodes according to the principle that different copies of the m data fragments are positioned in different available areas and the n coding fragments are stored in the P+1 available areas in a scattered manner.
6. A data read-write method, suitable for a storage service node in a cloud storage system, the method comprising:
receiving data storage layout information required by reading and writing a cloud storage disk and transmitted by a persistence management node in the cloud storage system, wherein the data storage layout information is generated by the persistence management node according to disaster tolerance requirement information of the cloud storage disk across an available area;
responding to a read-write request initiated by a computing node to the cloud storage disk, and performing data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information;
The cloud storage disk spans N available areas and adopts a redundancy storage mode of fusion of multiple copies and erasure codes, wherein the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation, N is more than or equal to 2, and N is a positive integer;
the disaster recovery requirement information of the cross-available area comprises the number f of abnormal fragments which can be tolerated when M available areas in N available areas are abnormal; the step of generating the data storage layout information includes:
determining the number N of coded slices and the number P of copies according to the number N of the available areas and the number f of abnormal slices; selecting K persistent storage nodes from at least P available areas according to the number P of fragments to be stored; according to at least the principle that different copies of m data fragments are positioned in different available areas, establishing a storage mapping relation between P x m+n fragments and K persistence storage nodes as data storage layout information required for reading and writing the cloud storage disk; wherein M is more than or equal to 1 and less than or equal to N, f is more than or equal to 1, P is more than or equal to K is more than or equal to (P+n), P is more than or equal to 2 and less than or equal to (N-1), N is more than or equal to f, M and N are respectively the number of data fragments and coding fragments in erasure codes, the number M of the data fragments is preset, P represents the number of copies of the M data fragments, and M, f, P, K, M and N are both positive integers.
7. The method of claim 6, wherein the step of providing the first layer comprises,
responding to a read-write request initiated by a computing node to the cloud storage disk, and performing data read-write operation on the persistent storage nodes in the cross-available area according to the data storage layout information, wherein the data read-write operation comprises the following steps:
responding to a write request initiated by a computing node to the cloud storage disk, and acquiring data to be stored;
performing slicing and erasure code coding processing on the data to be stored by adopting the number m of data slices and the number n of coding slices in erasure code to obtain m data slices and n coding slices;
and storing the P.m data fragments and the n coding fragments into the K persistence storage nodes according to a storage mapping relation between the P.m+n fragments and the K persistence storage nodes included in the data storage layout information.
8. The method of claim 7, wherein storing P x m data slices and n code slices into the K persistent storage nodes according to a storage mapping relationship between P x m+n slices and the K persistent storage nodes included in the data storage layout information comprises:
storing the same copy of the m data fragments into a persistent storage node located in the same available area in the K persistent storage nodes, and storing P different copies into the persistent storage nodes in P different available areas;
And the n code fragments are stored in at least one of the K persistent storage nodes in a scattered manner.
9. The method according to any one of claims 7-8, further comprising:
receiving storage success information returned by the K persistence storage nodes after successful data fragmentation or code fragmentation is stored respectively; recording a mapping relation between first LBA address information and first file address information of the data to be stored in the cloud storage disk according to the storage success message, wherein the first file address information corresponds to physical address information of the fragments corresponding to the data to be stored on the K persistent storage nodes;
and/or
Monitoring the running states of available areas of the K persistent storage nodes; and when the abnormal first available area is detected, storing the first data fragments and/or the first coding fragments which are required to be stored in the first available area into other available areas, and after the first available area is restored, migrating the first data fragments and/or the first coding fragments into a persistent storage node in the first available area.
10. The method according to any one of claims 6-8, wherein in response to a read-write request initiated by the computing node to the cloud storage disk, performing a data read-write operation on persistent storage nodes across available areas according to the data storage layout information, comprises:
Responding to a read request initiated by the computing node to the cloud storage disk, and acquiring second logic block addressing LBA address information of data to be read in the cloud storage disk;
determining second file address information according to the second LBA address information and a pre-recorded mapping relation between each LBA address information and file address information in the cloud storage disk;
according to the second file address information and the data storage layout information, physical address information of at least one data fragment corresponding to the data to be read on K persistence storage nodes is obtained;
and reading the at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to the physical address information of the at least one data fragment corresponding to the data to be read on the K persistent storage nodes.
11. The method of claim 10, wherein reading the at least one data fragment corresponding to the data to be read from the corresponding persistent storage node according to physical address information of the at least one data fragment corresponding to the data to be read on the K persistent storage nodes, comprises:
if the number of the abnormal available areas is less than or equal to M and the number of the abnormal fragments in the non-abnormal available areas is less than or equal to f, if the local available area where the storage service node is located is normal, reading at least one data fragment from the persistent storage node in the local available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the local available area;
If the local available area is abnormal and at least one data fragment corresponding to the data to be read in the non-abnormal available area is normal, reading the at least one data fragment from the persistent storage node in the non-abnormal available area according to the physical address information of the at least one data fragment corresponding to the data to be read on the persistent storage node in the non-abnormal available area;
if the local available area is abnormal and the abnormal data fragments exist in at least one data fragment corresponding to the data to be read in the non-abnormal available area, reading m normal fragments from the non-abnormal available area, and recovering the abnormal data fragments according to the m normal fragments; and acquiring at least one data fragment corresponding to the data to be read from the m normal fragments and the recovered abnormal fragments.
12. A data read-write method, suitable for a storage service management node in a cloud storage system, the method comprising:
creating a cloud storage disk which spans N available areas for a computing node, wherein N is more than or equal to 2, and N is a positive integer; the cloud storage disk adopts a redundancy storage mode of fusion of multiple copies and erasure codes, and the redundancy storage mode comprises coding fragmentation and multiple copy data fragmentation;
A storage service node is distributed for the cloud storage disk, so that the storage service node performs data read-write operation on the persistent storage nodes in the cross-available area according to data storage layout information, wherein the data storage layout information is generated by a persistent management node in a cloud storage system according to the cross-available area disaster recovery requirement information of the cloud storage disk; the disaster recovery requirement information of the cross-available area comprises the number f of abnormal fragments which can be tolerated when M available areas in N available areas are abnormal;
the step of generating the data storage layout information includes:
determining the number N of coded slices and the number P of copies according to the number N of the available areas and the number f of abnormal slices; selecting K persistent storage nodes from at least P available areas according to the number P of fragments to be stored; according to at least the principle that different copies of m data fragments are positioned in different available areas, establishing a storage mapping relation between P x m+n fragments and K persistence storage nodes as data storage layout information required for reading and writing the cloud storage disk; wherein M is more than or equal to 1 and less than or equal to N, f is more than or equal to 1, P is more than or equal to K is more than or equal to (P+n), P is more than or equal to 2 and less than or equal to (N-1), N is more than or equal to f, M and N are respectively the number of data fragments and coding fragments in erasure codes, the number M of the data fragments is preset, P represents the number of copies of the M data fragments, and M, f, P, K, M and N are both positive integers.
13. An electronic device, comprising: a memory and a processor; the memory is used for storing a computer program; the processor being coupled to the memory for executing the computer program for implementing the steps of the method of any one of claims 4-5, claims 6-11 and claim 12.
14. A computer readable storage medium storing a computer program, which when executed by a processor causes the processor to carry out the steps of any one of the methods of claims 4-5, claims 6-11 and claim 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311484532.2A CN117240873B (en) | 2023-11-08 | 2023-11-08 | Cloud storage system, data reading and writing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311484532.2A CN117240873B (en) | 2023-11-08 | 2023-11-08 | Cloud storage system, data reading and writing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117240873A CN117240873A (en) | 2023-12-15 |
| CN117240873B true CN117240873B (en) | 2024-03-29 |
Family
ID=89095039
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311484532.2A Active CN117240873B (en) | 2023-11-08 | 2023-11-08 | Cloud storage system, data reading and writing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117240873B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307221A (en) * | 2011-03-25 | 2012-01-04 | 国云科技股份有限公司 | Cloud storage system and implementation method thereof |
| CN106603673A (en) * | 2016-12-19 | 2017-04-26 | 上海交通大学 | Fine-grained cloud storage scheduling method based on erasure codes |
| CN109117292A (en) * | 2017-06-22 | 2019-01-01 | 中兴通讯股份有限公司 | A kind of method and apparatus and cluster storage system of cluster-based storage |
| CN110347344A (en) * | 2019-07-19 | 2019-10-18 | 北京计算机技术及应用研究所 | It is a kind of that block storage method is automatically configured based on distributed memory system |
| CN110990189A (en) * | 2019-11-29 | 2020-04-10 | 北京金山云网络技术有限公司 | Data storage method and device, electronic equipment and computer readable storage medium |
| CN113326006A (en) * | 2021-06-17 | 2021-08-31 | 上海天玑科技股份有限公司 | Distributed block storage system based on erasure codes |
| CN113626649A (en) * | 2021-08-02 | 2021-11-09 | Oppo广东移动通信有限公司 | Data storage method, data storage device, storage medium and electronic equipment |
| CN113901024A (en) * | 2021-09-26 | 2022-01-07 | Oppo广东移动通信有限公司 | Data storage system, data storage method, readable medium, and electronic device |
| CN114415976A (en) * | 2022-03-28 | 2022-04-29 | 深圳市杉岩数据技术有限公司 | Distributed data storage system and method |
| US11334456B1 (en) * | 2019-01-18 | 2022-05-17 | EMC IP Holding Company LLC | Space efficient data protection |
| WO2022198429A1 (en) * | 2021-03-23 | 2022-09-29 | 深圳市欢太科技有限公司 | Data redundancy backup method and apparatus, storage medium, and electronic device |
| CN115955488A (en) * | 2023-03-10 | 2023-04-11 | 智者四海（北京）技术有限公司 | Distributed storage copy cross-computer room placement method and device based on copy redundancy |
| CN116257186A (en) * | 2023-02-28 | 2023-06-13 | 济南浪潮数据技术有限公司 | Data object erasure code storage method, device, equipment and medium |
| CN116635831A (en) * | 2021-10-28 | 2023-08-22 | 华为云计算技术有限公司 | Data storage method and device and related equipment |
| CN116737466A (en) * | 2023-08-15 | 2023-09-12 | 中移(苏州)软件技术有限公司 | Backup processing method, device, system, electronic equipment and readable storage medium |
-
2023
- 2023-11-08 CN CN202311484532.2A patent/CN117240873B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307221A (en) * | 2011-03-25 | 2012-01-04 | 国云科技股份有限公司 | Cloud storage system and implementation method thereof |
| CN106603673A (en) * | 2016-12-19 | 2017-04-26 | 上海交通大学 | Fine-grained cloud storage scheduling method based on erasure codes |
| CN109117292A (en) * | 2017-06-22 | 2019-01-01 | 中兴通讯股份有限公司 | A kind of method and apparatus and cluster storage system of cluster-based storage |
| US11334456B1 (en) * | 2019-01-18 | 2022-05-17 | EMC IP Holding Company LLC | Space efficient data protection |
| CN110347344A (en) * | 2019-07-19 | 2019-10-18 | 北京计算机技术及应用研究所 | It is a kind of that block storage method is automatically configured based on distributed memory system |
| CN110990189A (en) * | 2019-11-29 | 2020-04-10 | 北京金山云网络技术有限公司 | Data storage method and device, electronic equipment and computer readable storage medium |
| WO2022198429A1 (en) * | 2021-03-23 | 2022-09-29 | 深圳市欢太科技有限公司 | Data redundancy backup method and apparatus, storage medium, and electronic device |
| CN113326006A (en) * | 2021-06-17 | 2021-08-31 | 上海天玑科技股份有限公司 | Distributed block storage system based on erasure codes |
| CN113626649A (en) * | 2021-08-02 | 2021-11-09 | Oppo广东移动通信有限公司 | Data storage method, data storage device, storage medium and electronic equipment |
| CN113901024A (en) * | 2021-09-26 | 2022-01-07 | Oppo广东移动通信有限公司 | Data storage system, data storage method, readable medium, and electronic device |
| CN116635831A (en) * | 2021-10-28 | 2023-08-22 | 华为云计算技术有限公司 | Data storage method and device and related equipment |
| CN114415976A (en) * | 2022-03-28 | 2022-04-29 | 深圳市杉岩数据技术有限公司 | Distributed data storage system and method |
| CN116257186A (en) * | 2023-02-28 | 2023-06-13 | 济南浪潮数据技术有限公司 | Data object erasure code storage method, device, equipment and medium |
| CN115955488A (en) * | 2023-03-10 | 2023-04-11 | 智者四海（北京）技术有限公司 | Distributed storage copy cross-computer room placement method and device based on copy redundancy |
| CN116737466A (en) * | 2023-08-15 | 2023-09-12 | 中移(苏州)软件技术有限公司 | Backup processing method, device, system, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117240873A (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106687911B (en) | Online data movement without compromising data integrity | |
| US8498997B2 (en) | Server image migration | |
| US11314687B2 (en) | Container data mover for migrating data between distributed data storage systems integrated with application orchestrators | |
| US10126946B1 (en) | Data protection object store | |
| US11614883B2 (en) | Distributed data storage system using erasure coding on storage nodes fewer than data plus parity fragments | |
| US20230308507A1 (en) | Commissioning and decommissioning metadata nodes in a running distributed data storage system | |
| CN108733311B (en) | Method and apparatus for managing storage system | |
| US20120151095A1 (en) | Enforcing logical unit (lu) persistent reservations upon a shared virtual storage device | |
| CN113672175A (en) | Distributed object storage method, device and equipment and computer storage medium | |
| JP2021524104A (en) | Master / Standby Container System Switching | |
| CN110704161B (en) | Virtual machine creation method and device and computer equipment | |
| CN111949210A (en) | Metadata storage method, system and storage medium in distributed storage system | |
| US20230418716A1 (en) | Anti-entropy-based metadata recovery in a strongly consistent distributed data storage system | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN104517067A (en) | Method, device and system for data access | |
| CN113687790A (en) | Data reconstruction method, device, equipment and storage medium | |
| JP2017531892A (en) | Improved apparatus and method for performing a snapshot of a block level storage device | |
| CN117240873B (en) | Cloud storage system, data reading and writing method, device and storage medium | |
| CN112711469A (en) | Cloud host migration method and device, computer equipment and storage medium | |
| CN109558082B (en) | Distributed file system | |
| CN116401004A (en) | Data sharing method, device, system and storage medium | |
| US10776045B2 (en) | Multiple data storage management with reduced latency | |
| CN117240917B (en) | Cache type cloud storage system, data read-write method, equipment and storage medium | |
| CN103559103A (en) | Data replication method and device | |
| CN116107515B (en) | Storage volume mounting and accessing method, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |