WO2023070935A1

WO2023070935A1 - Data storage method and apparatus, and related device

Info

Publication number: WO2023070935A1
Application number: PCT/CN2021/142795
Authority: WO
Inventors: 阿瓦鲁卡纳卡•库马尔; 库马尔潘卡吉; 钱纳巴斯帕雷努卡普拉萨德; 莫凯
Original assignee: 华为云计算技术有限公司
Priority date: 2021-10-28
Filing date: 2021-12-30
Publication date: 2023-05-04
Also published as: CN116635831A

Abstract

Embodiments of the present application provide a data storage method. The method can be applied to a data processing platform comprising a plurality of availability zones (AZs), and the data processing platform obtains a plurality of copies of data to be stored, and stores the plurality of copies of the data into different AZs of the data processing platform. In this way, when some of the AZs are unavailable due to natural disasters or other disasters, the disaster generally does not affect other AZs because a physical distance between different AZs is generally large, so that the data processing platform can continue to provide a data read-write service for a user on the basis of the copies of the data stored in other AZs running normally, thus the condition that the quality of the data read-write service provided by the data processing platform is reduced due to the occurrence of the disaster is avoided, and the data storage reliability of the data processing platform is improved. In addition, the embodiments of the present application further provide a corresponding apparatus and a related device.

Description

A data storage method, device and related equipment

This application claims the priority of the Indian patent application with the application number 202131049367 and the title of the invention "DATA MANAGEMENT SYSTEM AND METHOD" filed with the Indian Patent Office on October 28, 2021, the entire contents of which are incorporated herein by reference.

technical field

The embodiments of the present application relate to the technical field of databases, and in particular, to a data storage method, device, and related equipment.

Background technique

The data processing platform is used to provide users with data reading and writing services, such as data storage, data reading, etc., which may include distributed databases and file systems. Among them, a distributed database, such as an HBase database, usually includes a master (master) node and multiple partition server (Region Server, RS) nodes. In addition, the master node is used to assign to each RS node the region to which the data that the RS node is responsible for reading and writing belongs to. The number of partitions assigned to each RS node can be one or more; the RS node is used to allocate Write new data to the file system, or feed back the data in the text system requested by the user to the user, so as to realize the data reading and writing service.

In actual application, disasters such as natural disasters may reduce the quality of the data reading and writing services provided by the data processing platform for users, thus affecting the user experience. Therefore, there is an urgent need for a data storage method, so that when a disaster occurs, the quality of data reading and writing services provided by the data processing platform to users can be maintained at a relatively high level.

Contents of the invention

In view of this, the embodiment of the present application provides a data storage method, so that when a disaster occurs, the quality of data reading and writing services provided by the data processing platform for users can be maintained at a relatively high level. The present application also provides corresponding apparatuses, computing devices, computer-readable storage media, and computer program products.

In the first aspect, the embodiment of the present application provides a data storage method, which can be applied to a data processing platform including multiple availability zones AZ, and when storing data, the data processing platform can first obtain multiple copy, and store multiple copies of the data to be stored in different AZs of the data processing platform.

Since the data processing platform stores multiple copies of the data to be stored in different AZs, when some AZs are unavailable due to natural disasters or other disasters, the physical distance between different AZs is usually large, therefore, The disaster usually does not affect other AZs, so the data processing platform can continue to provide users with data read and write services based on the copies of the data to be stored in other AZs that are running normally, thereby avoiding disasters and reducing the data processing platform. The quality of data read and write services. Moreover, based on this data storage method, the allowable service interruption time length of the data processing platform can be 0, that is, the recovery time target reaches 0; the maximum data loss that the data processing platform can tolerate can reach 0, that is, the recovery point target Can be 0.

In addition, the data processing platform stores multiple copies of the data to be stored in different AZs instead of in one AZ. This makes it impossible for all copies of the data to be stored to be lost after some AZs become unavailable. In the event of data loss, the reliability of data stored on the data processing platform can be improved.

In a possible implementation manner, the data to be stored includes the target data and/or the partition to which the target data belongs, then, when the data processing platform stores multiple copies of the data to be stored on the data processing platform, specifically, the Multiple copies of the target data are stored in different AZs of the data processing platform, and/or multiple partition copies of the partition to which the target data belongs are stored in different AZs of the data processing platform. In this way, when some AZs are unavailable, the data processing platform can obtain target data copies and/or partition copies from other AZs, so as to continue to provide users with data read and write services using the target data copies and/or partition copies in other AZs , to improve the reliability of data reading and writing services provided by the data processing platform.

In a possible implementation manner, the data to be stored includes target data and the partition to which the target data belongs, and the data processing platform includes a distributed database and a file system, wherein the distributed database includes partition servers RS under multiple AZs node, the file system includes data nodes under multiple AZs, when the data processing platform stores multiple copies of the data to be stored in different AZs, it may specifically store multiple partition copies in different AZs in the distributed database The RS node under , stores multiple data copies to the data nodes under different AZs in the file system. In this way, the reliability of data reading and writing services provided by the data processing platform can be improved.

Among them, multiple AZs under the distributed database and multiple AZs under the file system may overlap (all or part of the AZs may be the same), for example, the distributed database and the file system may have the same multiple AZs. Alternatively, the multiple AZs under the distributed database do not overlap with the multiple AZs under the file system, for example, the distributed database includes AZ1 to AZ5, and the file system includes AZ6 to AZ10. In this way, the reliability of data reading and writing services provided by the data processing platform can be further improved.

In a possible implementation, when the data processing platform stores multiple copies of data in data nodes under different AZs in the file system, it can first obtain the physical distance between different AZs in the file system, and according to the file The physical distance between different AZs in the system, determine the multiple first AZs in the file system, and the determined physical distance between the multiple first AZs does not exceed the distance threshold (such as 40 kilometers, etc.), so that the data processing The platform can store multiple data copies to the data nodes under the multiple first AZs. In this way, when some AZs fail and the data processing platform reads data copies from other AZs, since the other AZs are relatively close to the AZ, the data processing platform can quickly read the data copies, so that the data copy can be obtained Latency is kept low.

In a possible implementation manner, when the data processing platform stores multiple copies of data in data nodes under different AZs in the file system, it can obtain the availability of each AZ in the file system, for example, through The ratio between the data nodes available in the AZ and all data nodes is determined, so that the data processing platform can store multiple data copies in multiple first AZs in the file system according to the availability of each AZ in the file system data nodes, and the file system further includes at least one second AZ with a lower degree of availability, wherein the lower degree of availability of the second AZ means that the degree of availability of the second AZ is lower than that of the first AZ. In this way, the data processing platform can preferentially select the data nodes under the first AZ with higher availability to store the data copy, so as to improve the reliability of the data processing platform for reading and writing data.

In a possible implementation manner, when the data processing platform stores multiple copies of data in data nodes under different AZs in the file system, it can obtain the availability of each AZ in the file system, for example, through The ratio between the data nodes available in the AZ and all data nodes is determined, so that the data processing platform can store some of the data copies in multiple copies of the data in multiple copies of the file system according to the availability of each AZ in the file system. data nodes under the first AZ, and the file system also includes at least one second AZ, and the availability of the second AZ is lower than that of the first AZ; when the availability of the at least one second AZ rises to When the threshold is preset, the data processing platform may store other data copies in the multiple data copies to the data nodes under the at least one second AZ. In this way, when storing multiple data copies, it is possible to avoid migrating the data storage tasks required by AZs with low availability to other AZs, so as to avoid increasing the load of other AZs.

In a possible implementation manner, when storing multiple partition copies to RS nodes under different AZs in the distributed database, it may be specifically to obtain allocation indication information for the multiple partition copies, and the allocation indication information is used for Indicates the ratio of the number of copies of multiple partition copies stored in different AZs, so that the data processing platform can store multiple partition copies to RS nodes under different AZs in the distributed database according to the allocation instruction information. Wherein, the allocation indication information may be pre-configured by technicians or users, or the allocation indication information may be automatically generated by the data processing platform, and the like. In this way, the data processing platform can implement cross-AZ storage of multiple partition copies according to the allocation indication information.

In a possible implementation, the allocation indication information can be determined according to the load of the RS nodes under each AZ in the distributed database. In this way, when the data processing platform stores multiple partition copies according to the allocation indication information, it can balance multiple AZ load.

In a possible implementation manner, the distributed database and the file system both include multiple target AZs, and the multiple target AZs have already stored multiple partition copies of the partition to which the target data belongs, then when storing multiple data copies, It can track multiple target AZs that store multiple partition copies, and store multiple data copies to data nodes under different target AZs. In this way, when providing data read and write services, the distributed database can read the target data from the local (that is, the AZ where the partition copy is located) data node based on the partition copy, thereby reducing the delay in reading data and improving the data quality. The efficiency with which the processing platform feeds back data to users.

In a possible implementation, after storing multiple partition copies to RS nodes under different AZs in the distributed database, the data processing platform can perform load balancing on the RS nodes under different AZs in the distributed database to adjust The number of partition replicas stored in different AZs. In this way, the load balancing of different RS nodes is realized, and the excessive load of some ES nodes is prevented from affecting the quality of data reading and writing services provided by the data processing platform as a whole.

In the second aspect, based on the same inventive concept as the method embodiment in the first aspect, the embodiment of the present application provides a data storage device. The device has functions corresponding to each implementation manner for realizing the above-mentioned first aspect. This function may be implemented by hardware, or may be implemented by executing corresponding software on the hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a third aspect, the present application provides a computing device, where the computing device includes a processor, a memory, and a display. The processor and the memory communicate with each other. The processor is configured to execute instructions stored in the memory, so that the computing device executes the data storage method in the first aspect or any implementation manner of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computing device, the computing device executes the above-mentioned first aspect or any of the first aspects. A data storage method described in an implementation manner.

In a fifth aspect, the present application provides a computer program product containing instructions, which, when run on a computing device, causes the computing device to execute the data storage method described in the first aspect or any implementation manner of the first aspect .

On the basis of the implementation manners provided in the foregoing aspects, the present application may further be combined to provide more implementation manners.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some implementations recorded in the application. For example, those skilled in the art can also obtain other drawings based on these drawings.

FIG. 1 is a schematic diagram of the architecture of an exemplary data processing platform 100 of the present application;

FIG. 2 is a schematic structural diagram of a cluster 200 constructed across availability zones;

FIG. 3 is a schematic flow diagram of a data storage method provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of storing multiple data copies provided by the embodiment of the present application;

FIG. 5 is a schematic structural diagram of a data storage device provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a hardware structure of a computing device provided by an embodiment of the present application.

Detailed ways

The solutions in the embodiments provided in the present application will be described below with reference to the drawings in the present application.

The terms "first", "second" and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is merely a description of the manner in which objects with the same attribute are described in the embodiments of the present application.

As shown in FIG. 1 , it is a schematic structural diagram of an exemplary data processing platform 100 . As shown in FIG. 1 , the data processing platform 100 includes a distributed database 101 and a file system 102 . Wherein, the file system 102 can be used to persistently store data in the form of files, and the distributed database 101 can be used to manage the data in the file system 102, including reading, writing, and merging of data.

The file system 102 includes multiple data nodes (datanodes), and different data nodes may belong to different availability zones (availability zones, AZ). In FIG. 1 , the file system 102 includes data nodes 1021 to 1024 as an example for illustration, and data nodes 1021 and 1022 belong to AZ1, and data nodes 1023 and 1024 belong to AZ2. Among them, AZ usually refers to a collection of one or more physical data centers, with independent wind, fire, water and electricity, and computing, network, storage and other resources can be logically divided into multiple clusters within the AZ. As some examples, the file system 102 may be, for example, a distributed file system (distributed file system, DFS), a Hadoop distributed file system (hadoop distributed file system, HDFS), etc., which are not limited in this embodiment. In practical applications, the file system 102 may also include a named node (namenode), the named node (not shown in FIG. 1 ) may also be called a master node, and is used to manage multiple data nodes, including namespace, metadata that records the data stored in each data node, etc.

The distributed database 101 includes a master node 1011 and multiple partition server (region server, RS) nodes, and different RS nodes belong to different availability zones. In FIG. 1 , the distributed database 101 includes the RS node 1012 and the RS node 1012 is taken as an example for illustration, and the RS node 1012 belongs to AZ1, and the RS node 1012 belongs to AZ2.

The master node 1011 is used to divide the data managed by the distributed database 101 (that is, the data stored in the file system 102) to obtain multiple partitions, each partition includes one or more data identifiers, and belongs to Data in different partitions often differs. As an implementation example of partitioning, when managing each piece of data in the distributed database 101, part of the content in the piece of data can be used as a primary key (primary key) corresponding to the piece of data, and the primary key is used for This piece of data is uniquely identified in the distributed database 101, so that the primary node 1011 can perform interval division according to the possible value range of the primary key, and each divided interval corresponds to a partition. For example, assuming that the value range of the primary key in the distributed database 101 is [0, 1000000], then the primary node 101 can divide the value range of the primary key into 100 intervals, which are respectively [0, 10000), [10000, 20000), ..., [980000, 990000), [990000, 1000000], each partition can be used to index 10,000 pieces of data, correspondingly, based on the 100 partitions, the distributed database 101 can manage 1 million Article data. Further, the master node 1011 can also achieve high availability through a distributed application coordination service (such as zookeeper service, etc.) 1014, as shown in FIG. 1 .

Meanwhile, the master node 1011 is also used to allocate partitions for the RS nodes 1012 and 1013 , and the partitions assigned to each RS node can be maintained through the management table created by the master node 1011 . RS node 1012 and RS node 1013 are respectively used to perform data read and write services belonging to different partitions. As shown in Figure 1, RS node 1012 performs data read and write services belonging to partition 1 to partition N, while RS node 1013 performs data read and write services belonging to partition N+1. Data read and write services to partition M.

Exemplarily, the master node 1011, the RS node 1012 and the RS node 1013 can all be implemented by hardware or software. For example, when the master node 1011 and each RS node are implemented by hardware, both the master node 1011 and multiple RS nodes may be physical servers in the distributed database 101 . That is, during actual deployment, at least one server in the distributed database 101 may be configured as the master node 1011, and other servers in the distributed database 101 may be configured as RS nodes. Alternatively, the master node 1011 and each RS node are implemented by software. At this time, the master node 1011 and multiple RS nodes may be processes or virtual machines running on one or more devices (such as servers, etc.).

It should be noted that the data processing platform 100 shown in FIG. 1 is only used as an exemplary illustration, and is not used to limit the specific implementation of the data processing platform. For example, in other possible data processing platforms 100, the distributed database 101 may include any number of master nodes and RS nodes, or the RS nodes in the distributed database 101 and the data nodes in the file system 102 may belong to different AZ, which is not limited in this application.

The distributed database 101 can be respectively connected with the file system 102 and the client 103, for example, it can be connected through a wireless communication protocol such as a hypertext transfer protocol (HyperText Transfer Protocol, HTTP). Assuming that the client 103 has a need to modify data or write new data to the RS node 1012, the client 103 can send a data write request to the RS node 1012, and the data write request carries the data to be written and the corresponding data processing Operation content (such as write operation, modification operation, etc.). Then, the RS node 1012 can analyze and receive the data write request, generate a corresponding data processing record based on the data to be written and the data processing operation, and write the data processing record into a pre-created write ahead log (write ahead log, WAL) file. After determining that the writing of the WAL file is successful, the RS node 1012 persistently stores the WAL file to the file system 102 . And, the RS node 1012 inserts the data to be written in the data processing record into the internal memory 1 of the RS node 1012 . For example, the RS node 1012 can first determine the primary key corresponding to the data processing record, and determine which partition to write the data processing record into according to the partition interval to which the value of the key belongs, so that the RS node 1012 can process the data The data to be written in the record is inserted into the storage area corresponding to the partition in the memory 1; then, the RS node 1012 can feed back to the client 103 that the data writing is successful.

Normally, the RS node 1012 will write data in the memory 1 for one or more clients 103, so the amount of data temporarily stored in the memory 1 will continue to increase. When the amount of data in the memory 1 reaches the preset threshold, the RS node 1012 can persistently store the data in the memory 1 to the file system 102, such as persistently storing the data in the memory 1 to the data node 1021 under AZ1.

Further, the RS node 1012 is also configured with a region store file (region store files) for each partition, and after persistently storing data in the file system 102, the RS node 1012 can store the data of each partition in the file system 102 The files stored in the partition are added to the partition storage file corresponding to the partition. Specifically, the file name corresponding to each data in the partition can be added in the directory of the partition storage file.

Correspondingly, when the client 103 needs to read data, it can send a data read request to the RS node 1012, and the data read request carries the primary key of the data to be read. After receiving the data write request, RS node 1012 can determine the partition according to the value of the primary key corresponding to the data to be written, so that RS node 1012 can store files from the data nodes under AZ1 according to the partition storage file corresponding to the partition. Find out the data required by the client 102 and feed it back to the client 103 .

When the data processing platform 100 is deployed, the data processing platform 100 can be used as a local resource to provide local data reading and writing services to clients accessing the data processing platform 100 through the distributed database 101 and the file system 102 . Alternatively, the data processing platform 100 can also be deployed on the cloud, and at this time, the distributed database 101 and the file system 102 can provide cloud services for reading and writing data to clients connected to the cloud.

In actual application scenarios, disasters such as natural disasters will inevitably occur, and the disaster may cause some AZs (such as AZ1 or AZ2) to be unavailable, for example, the disaster causes physical damage to some or all computing devices under the AZ. At this time, the data stored in the partitions in the RS nodes under the AZ and/or in the data nodes under the AZ may be lost or unreadable due to the unavailability of the AZ, which will reduce the data processing platform 100. The quality of data reading and writing services, that is, it is difficult for users to obtain the data stored in the AZ, which affects the user experience.

Based on this, the embodiment of the present application provides a data storage method, aiming at allocating partitions and/or copies of data to different AZs, so as to improve the reliability of data reading and writing services provided by the data processing platform 100 . During specific implementation, for the data to be stored, the data processing platform 100 may acquire multiple copies of the data to be stored, and store the multiple copies of the data to be stored in different AZs of the data processing platform 100 . Exemplarily, the data to be stored may be data newly written by the user and/or the partition to which the data belongs, for example. Since the data processing platform 100 replicates the data to be stored, and different copies of the data to be stored are stored in different AZs, when AZ1 (or AZ2) is unavailable due to natural disasters or other disasters, due to different The physical distance between the AZs is usually relatively large. Therefore, although the disaster makes AZ1 unavailable, it usually does not affect AZ2, so that the data processing platform 100 can be based on the data to be stored stored in AZ2 (or AZ1) in normal operation. continue to provide data reading and writing services for users, thereby avoiding disasters and reducing the quality of data reading and writing services provided by the data processing platform 100. Moreover, the allowable service interruption time length of the data processing platform 100 can be 0, that is, the recovery time objective (RTO); the maximum amount of data loss that the data processing platform 100 can tolerate can reach 0, that is, the recovery point objective (recovery point objective, RPO) can be 0.

In addition, the data processing platform 100 stores multiple copies of the data to be stored in different AZs, rather than storing them in one AZ. This makes it impossible for all copies of the data to be stored to be stored after some AZs become unavailable. All data loss occurs, so that the reliability of data storage on the data processing platform 100 can be improved.

Further, when the data to be stored is new data provided by the user and the partition to which the new data belongs, the distributed database 101 obtains the partition to which the data belongs during the process of storing multiple copies of the data to be stored by the data storage platform 100 Multiple copies of partitions, such as obtaining multiple copies of partitions by copying partitions. Then, the distributed database 101 stores multiple partition copies to RS nodes under different AZs included in the distributed database 101, such as storing part of the partition copies to the RS node 1012 under AZ1, and storing the remaining partition copies to AZ2 RS node 1013 . In addition, the file system 102 also obtains multiple data copies of the data, for example, the distributed database 101 sends multiple data copies obtained by duplicating the data to the file system 102, or the file system 102 sends the distributed database 101 The sent data is copied to obtain multiple copies of the data, etc. Then, the file system 102 stores multiple data copies to data nodes under different AZs included in the file system 102. For example, part of the data copies are stored in the data nodes under AZ1, and the remaining data copies are stored in the data nodes under AZ2. .

Since the data processing platform 100 stores multiple copies of data in different AZs rather than in one AZ, this will not cause data loss for all data copies of one piece of data when some AZs become unavailable. Therefore, the reliability of data stored by the data processing platform 100 can be improved. Similarly, the data processing platform 100 stores multiple partition copies in different AZs, so that when some AZs become unavailable, data loss will not occur in all partition copies corresponding to one piece of data, so that the data processing platform can 100 The quality of data read based on partition replicas is maintained at a high level.

In actual application, based on the data processing platform 100 shown in FIG. 1 , computing devices (including RS nodes, data nodes, etc.) under multiple AZs can be deployed into clusters distributed across AZs, and the data processing platform 100 can utilize the cluster Provide reliable data read and write services. As an example, the deployed cluster can be shown in FIG. 2 , and the cluster 200 can include computing devices under AZ1 and AZ2. Specifically, the cluster 200 can include RS nodes 1012, data nodes 1021, and data nodes 1022 under AZ1. In addition, it also includes other nodes under the AZ1, such as a named node 1031 for updating the namespace of the cluster 200, recording metadata of data stored in the data node, a log node (journalnode) 1032 for storing and managing logs, A resource management device 1033 for managing resources in AZ1, a node management device 1034 for managing data nodes, a node management device 1035, and a master server 1036 for monitoring and managing RS nodes 1012 (such as Hmaster etc.); Moreover, a distributed application program coordination service may also be configured in the AZ1 to improve the high availability of the AZ1. Similarly, AZ2 and AZ1 also have similar configurations, as shown in Figure 2 for details. Wherein, the computing device under AZ1 and the computing device under AZ2 can be active and standby each other. For example, when the named node 1031 under AZ1 fails, the named node 1041 under AZ2 can update the namespace of the cluster 200, etc.

It should be noted that the cluster 200 shown in FIG. 2 is based on the deployment of the data processing platform 100 shown in FIG. For each computing device under AZ3 shown by the dotted line, this embodiment does not limit the specific architecture of the cluster deployed across AZs. Moreover, multiple clusters may be deployed across AZs in the data processing platform 100, and different clusters include computing devices under different AZs. For example, assuming that the data processing platform 100 includes 6 AZs, namely AZ1 to AZ6, one cluster may be deployed based on AZ1 to AZ3, and another cluster may be deployed based on AZ4 to AZ6, which is not limited in this embodiment.

In addition, in a further possible implementation manner, a standby cluster 300 may also be deployed for the cluster 200, so as to replicate and retrieve data stored in the cluster 200 (including partitions, data written in the file system 102, etc.) storage, so that the reliability of data read and write services provided by the cluster 200 can be further improved. Exemplarily, an asynchronous replication method may be adopted between the cluster 200 and the cluster 300 to replicate data in the cluster 200 to the cluster 300 .

Next, various non-limiting specific implementations of data storage are described in detail.

Referring to FIG. 3 , it is a schematic flowchart of a data storage method in an embodiment of the present application. The method can be applied to the data processing platform 100 shown in FIG. 1 above, and specifically can be executed by the distributed database 101 and the file system 102 . Alternatively, the method may also be executed by a device separately configured in the data processing platform 100, which is not limited in this embodiment. For ease of description, the data to be stored includes target data (such as the new data provided by the above-mentioned user) and the partition of the target data, and the distributed database 101 and the file system 102 execute the data storage method as an example. The data storage method shown in Figure 3 may specifically include:

S301: The distributed database 101 acquires multiple data copies of the target data, and multiple partition copies of the partition to which the target data belongs.

The target data acquired by the distributed database 101 may be, for example, new data provided by the user to the data processing platform 100, or data generated by the data processing platform 100 based on the user's modification operation on the data. Usually, the data processing platform 100 can write the primary key contained in the target data into the partition, so that the target data can be managed subsequently based on the primary key recorded in the partition; and, the data processing platform 100 can also write the The target data is persistently stored in the file system 102 .

In this embodiment, the distributed database 101 can respectively replicate the target data and the partition to which the target data belongs, so that multiple data copies of the target data can be obtained (the target data itself can also be regarded as a data copy), a partition copy (The partition to which the target data belongs can also be regarded as a partition copy).

It is worth noting that, in this embodiment, the distributed database 101 is used as an example to replicate partitions. In other practical applications, the distributed database 101 executes the replication operation on the partition to which the target data belongs to obtain multiple partition copies, and The target is sent to the file system 102, so that the file system 102 executes the copy operation of the target data to obtain multiple data copies. This embodiment does not limit it.

S302: The distributed database 101 stores multiple partition copies to RS nodes under different AZs included in the distributed database 101.

If the distributed database 101 stores all partition copies of the target data in one AZ, then when the AZ is unavailable due to a disaster, it is difficult for the distributed database 101 to use the partition copies in the AZ to provide users with data for the target data Read and write services. Therefore, in this embodiment, the distributed database 101 stores multiple partition copies in at least two AZs, so that even if one of the AZs is unavailable, the distributed database 101 can also manage the target through the partition copies stored in the remaining AZs. data. In this way, unreadable target data in the data processing platform 100 due to the unavailability of a single AZ can be avoided, and data fault tolerance at the AZ level can be realized.

Wherein, the number of partition copies allocated to different AZs may be the same or different. For example, when the distributed database 101 includes 3 AZs and the number of partition copies to which the target data belongs is 3, the distributed database 101 can store the 3 partition copies in the 3 AZs respectively, and each AZ stores a copy of the partition. Alternatively, when the distributed database 101 stores three partition copies in AZ1 and AZ2 (not in AZ3), one partition copy can be stored in AZ1, two partition copies can be stored in AZ2, and so on.

Exemplarily, this embodiment provides the following four implementations for storing multiple partition copies in different AZs:

In the first possible implementation manner, the distributed database 101 may acquire the allocation indication information of the multiple partition copies, the allocation indication information may include the AZ identifier and the proportion of the number of copies, and may be used to indicate that the multiple partition copies The proportion of copies stored in different AZs. In this way, the distributed database 101 can store multiple partition copies to RS nodes under different AZs in the distributed database according to the allocation indication information. For example, suppose the availability zone includes AZ1, AZ2, and AZ3, and the number of partition replicas is 4, and the allocation indication information can be the allocation expression "REP: AZ1[0.5], AZ2[0.25], AZ3[0.25]", then According to the distribution expression, the distributed database 101 can store 2 (ie 0.5*4) partition copies in AZ1, store 1 (ie 0.25*4) partition copies in AZ2, and store 1 partition copy in AZ2 Stored in AZ3. It is worth noting that among the multiple partition copies written in multiple AZs in the distributed database 101, one of the partition copies is used as the primary partition copy, that is, the distributed database 101 usually provides data read and write for users based on the primary partition copy. service, while the rest of the partition copies are used as secondary partition copies to continue to provide users with data read and write services based on the secondary partition copies when the primary partition copy is unreadable or data loss occurs.

In practical applications, the distributed database 101 may process copies of multiple partitions in batches based on the above allocation indication information. For example, assuming that there are currently 10 partitions, and the number of copies of each partition is 4, then when the allocation indication information is "REP: AZ1[0.5], AZ2[0.25], AZ3[0.25]", the distributed database 101 can Store 20 (that is, 0.5*4*10) copies of the partition in AZ1, store 10 (that is, 0.25*4*10) copies of the partition in AZ2, and store 10 copies of the partition in AZ3. Among them, 3 AZs store a copy of each partition.

Wherein, the allocation indication information may be determined by the distributed database 101 according to the load of the RS nodes in each AZ, for example. For example, when it is assumed that the load of RS node 1012 in AZ1 is 10%, the load of RS node 1013 in AZ2 is 30%, and the load of RS node 1014 in AZ3 is 35%, and the number of copies of each partition is set is 4, the distributed database 101 can determine that the allocation indication information is specifically "REP: AZ1[0.5], AZ2[0.25], AZ3[0.25]", that is, AZ1 is used to store 50% of the partition copies, and AZ2 and AZ3 are both used To store 25% of the partition copies, so as to balance the load among RS nodes under different AZs. Of course, the allocation instruction information may also be determined in other ways, such as manual configuration by a technician in advance, which is not limited in this embodiment.

As an implementation example, the distributed database 101 may include a main server (such as the main server 1036 in FIG. 2 ) and an AZ aware balancer (AZ aware balancer), wherein the main server can perceive effective RS nodes, such as Valid RS nodes can be determined based on heartbeat messages sent by RS nodes, so that the master server can generate a network topology map based on valid RS nodes. In this way, when the master server receives an allocation request for multiple partition copies generated by the distributed database 101, the master server can forward the allocation request to the AZ-aware balancer, and the AZ-aware balancer will use the AZ-based network The topology map (AZ based network topology), together with the above allocation instructions, determines the AZ used to store the partition replica. Further, the distributed database 101 can also be provided with a default balancer (default balancer), and, for some partitions that do not need to be replicated and stored (such as the importance of the partition is low, etc.), the distributed database 101 can also use this A default balancer is used to determine the AZ where the partition is stored. The default balancer may, for example, determine the AZ of the storage partition through a random algorithm or a load balancing strategy. Wherein, the main server, the AZ-aware equalizer, and the default equalizer can all be implemented by software (such as a process) or hardware (such as a separately configured computing device), which is not limited in this embodiment.

In the second possible implementation, for multiple partition copies of a partition, if the number of partition copies is greater than the number of AZs used to store the multiple partition copies, the distributed database 101 may first create AZ allocates a partition copy, and, for the remaining unassigned partition copies, the distributed database 101 can determine the AZ where the RS node with a relatively small load is located according to the current load of the RS nodes in each AZ, so that the distributed database 101 may allocate the remaining partition copies to the AZ for storage. In this way, the distributed database 101 can flexibly allocate storage locations of partition copies according to the load conditions of RS nodes under multiple AZs in various time periods, so as to realize load balancing of RS nodes.

In the third possible implementation manner, because some data nodes under the AZ may fail or the data storage load is too large in the actual application scenario, which affects the ability of the AZ to further store data, the distributed database 101 can be based on each AZ Availability, which determines the AZ where the replica of the partition is stored. Specifically, during the process of writing partition copies to multiple AZs, the distributed database 101 may first obtain the availability degree of each AZ for each AZ. The ratio between the total number of data nodes in the AZ is determined and can reflect the availability of the AZ. For example, assuming that the AZ includes 10 data nodes, and the number of available nodes is 8, the availability of the AZ may be 80% (ie, 8/10). Among them, the available data node refers to the data node with further normal ability to read and write data; correspondingly, when the data node is physically damaged or the data is read and written incorrectly, the data node can be determined to be faulty, that is, unavailable , or, when the amount of data stored in the data node is too large and the data node cannot further store data, the data node may also be determined as an unavailable data node. In practical applications, whether a data node is available can also be defined in other ways, for example, when the data to be stored does not have the permission to be written into the data node, the data node can be determined to be unavailable relative to the data to be stored, etc. Then, the distributed database 101 can determine a plurality of first AZs whose availability is higher than a preset threshold and at least one second AZ whose availability is lower than a preset threshold according to the availability of each AZ, so that the distributed database 101 can A plurality of partition copies are written to the plurality of first AZs. In this way, the number of new partition copies written by the distributed database 101 to the AZ with a low degree of availability can be reduced, thereby avoiding the excessive load of the RS node under the AZ with a low degree of availability, causing the RS node to be responsible for The amount of data read exceeds the maximum amount of data that can be stored in this AZ.

In the fourth possible implementation, when the distributed database 101 writes the partition copy to the AZ, some AZs may be less available or become unavailable. Based on this, the distributed database 101 can Suspend writing partition copies to this part of the AZ. Specifically, before the distributed database 101 writes a partition copy to one of the AZs, it may first obtain the availability of the AZ, and if the availability of the AZ is lower than the preset threshold, the distributed database 101 may not write to the AZ. Instead, the master server can create a cache queue for the AZ, such as a region in transaction (RIT) queue, etc., mark the partition copy as unallocated and write it to the cache queue middle. Then, the distributed database 101 continues to write the partition copy to the next AZ. In practical application, if the master server with the identity of "primary" is located in the AZ with a low degree of availability, the distributed database 101 can switch the identity of the master server in other AZs with a high degree of availability from "standby" to "primary". identity, and create a cache queue for that AZ. For an AZ that does not store a partition copy, the distributed database 101 can monitor the availability of the AZ, such as by configuring the RIT work (chore) node for monitoring, and if the availability of the AZ rises and exceeds the preset threshold, Then the distributed database 101 can write the partition copy in the cache queue corresponding to the AZ into the RS node belonging to the AZ. In practical applications, if the partition copy stored in the AZ with low availability is a secondary partition copy, the distributed database 101 may write the secondary partition copy to the AZ after the availability of the AZ increases. If the partition copy stored in the AZ with low availability is the primary partition copy, at this time, because the availability of the AZ is too low, the distributed database 101 can select a secondary partition copy from other AZs with high availability. The partition replica acts as the primary partition replica to ensure the high availability of the primary partition replica in the distributed database 101, and the partition replica in the AZ acts as the secondary partition replica.

It is worth noting that the above four implementation methods are only used as some exemplary illustrations. In practical applications, the distributed database 101 may also store multiple partition copies in different AZs based on other methods. For example, the distributed database 101 may By combining the above implementation methods, multiple AZs with high availability and close physical distances are selected to store multiple partition copies, etc.

S303: The file system 102 stores multiple data copies to data nodes under different AZs in the file system 102 .

Wherein, multiple data copies of the target data may be provided by the distributed database 101 to the file system 102, or the file system 102 may replicate the target data to obtain the multiple data copies. Similar to the storage partition copy, the file system 102 may store the obtained multiple data copies in different AZs, specifically, the data nodes in different AZs. Wherein, each AZ includes at least one data copy. In this way, even if one of the AZs where the data copy is stored is unavailable, the distributed database 101 can read the target data from the remaining AZs, so as to avoid loss of the target data and achieve AZ-level data fault tolerance.

Exemplarily, this embodiment provides the following four implementations for storing multiple copies of data in different AZs:

In the first possible implementation manner, similar to the above-mentioned distributed database 101 storing multiple partition copies, the file system 102 may also store multiple data copies based on allocation indication information. In this case, the allocation indication information may include AZ identification And the proportion of the number of copies, which can be used to indicate the proportion of the number of copies of the data nodes that store the multiple copies of data in different AZs. For example, assuming that the availability zone includes AZ1, AZ2, and AZ3, and the number of data copies is 4, and the allocation indication information can specifically be the allocation expression "REP: AZ1[0.5], AZ2[0.25], AZ3[0.25]", then According to the allocation expression, the distributed database 101 can store 2 (ie 0.5*4) copies of data in AZ1, store 1 (ie 0.25*4) copies of data in AZ2, and store 1 copy of data in AZ2 Stored in AZ3. In this way, the file system 102 can store multiple data copies to data nodes under different AZs in the file system 102 according to the allocation indication information. Wherein, the allocation instruction information may be determined according to the load of the data nodes under each AZ, or manually configured by a technician, which is not limited in this embodiment.

Exemplarily, a named node remote procedure call server (namenode remote procedure call server) and an availability zones block placement policy (availability zones block placement policy, AZ BPP) node may be set in the file system 102. When there is a replication task for the target data in the file system 102, the named node remote procedure call server can instruct the AZ BPP node to execute the replication process of the target data; the AZ BPP node can determine multiple AZs and each The number of copies of the target data stored in the AZ, so that the corresponding data storage process can be executed. Further, a default block placement policy (default block placement policy) node may also be set in the file system 102, and, for some data that does not need to be copied and stored (such as the importance of the data is low, etc.), the file system 102 may also The default block placement strategy node is used to determine the AZ for storing the data. The default block placement strategy node can be, for example, the AZ for storing data determined by a random algorithm or a load balancing strategy. Wherein, the file system 102 can generate an AZ based network topology based on the data nodes under each AZ, so that the default block placement policy node can determine the AZ for storing data copies according to the network topology.

In a second possible implementation manner, the file system 102 may determine an AZ for storing data copies according to physical distances between different AZs. Specifically, the file system 102 can obtain the physical distance between different AZs in the file system 102, and store multiple copies of data in multiple copies of the data in the file system 102 that are physically closer to each other according to the physical distance between different AZs. The data nodes under the first AZ, that is, the physical distance between each first AZ and at least one of the first AZs in the plurality of first AZs does not exceed the distance threshold. At the same time, the file system 102 also includes at least one data node under the second AZ, and the physical distance between the second AZ and each first AZ exceeds a distance threshold (such as 40 kilometers, etc.). In this way, when there is a fault in some first AZs, the data processing platform 100 can read copies of the target data from the remaining first AZs, and since the physical distances between different first AZs are relatively close, the data processing platform 100 The time delay for 100 to read data copies from other available first AZs is usually less than the time delay for reading data copies from the first AZ before the failure.

In the third possible implementation manner, because some data nodes under the AZ may fail or the data storage load is too large in the actual application scenario, which affects the ability of the AZ to further store data, the file system 102 can be based on each AZ Availability status, determine the AZ where the data copy is stored. Specifically, the file system 102 can first obtain the availability of each AZ, which can be determined by, for example, calculating the ratio between the number of data nodes available in the AZ and the total number of data nodes in the AZ, and can reflect AZ AVAILABILITY. After obtaining the availability of each AZ, the file system 102 can store multiple copies of data in the data nodes under the multiple first AZs with relatively high availability in the file system 102 according to the availability of each AZ, each At least one data copy is stored in the first AZ. At the same time, the file system 102 also includes at least one data node under the second AZ, and the availability of the second AZ is lower than that of the first AZ. In this way, the file system 102 can preferentially allocate the data copy to an AZ with a high degree of availability for storage, thereby improving the reliability of data reading and writing of the data processing platform 100 .

In the fourth possible implementation manner, when the file system 102 writes data copies to the AZ, some AZs may be less available or become unavailable. Based on this, the file system 102 can suspend Write a data copy to this part of AZ. Specifically, before writing multiple data copies to multiple AZs, the file system 102 can first obtain the availability of each AZ, and the file system 102 can write part of the multiple data copies according to the availability of each AZ The copy is stored in the data nodes under the first AZ with higher availability (higher than the preset threshold) in the file system 102, and at least one data copy is stored in each first AZ. As for at least one AZ in the file system 102 with a lower degree of availability (below a preset threshold), that is, a second AZ other than the first AZ, the file system 102 may temporarily not write a data copy to the second AZ. Then, when the availability of the second AZ rises to a preset threshold, the file system 102 stores the rest of the unstored data copies in the second AZ. In this way, when the file system 102 stores multiple data copies, it can avoid migrating the data storage task of the AZ with low availability to other AZs, so as to avoid increasing the load of other AZs.

For example, as shown in FIG. 4, for data A, the file system 102 may sequentially write three data copies of the data A into data node 1 under AZ1, data node 2 under AZ2, and data node 3 under AZ3; and When storing multiple copies of data corresponding to data B, after the file system 102 writes the copy of data B into AZ1, if it determines that the availability of AZ2 is low (such as unavailable, etc.), the file system 102 can suspend writing the copy of data B to AZ2. A copy of data B is written, while a copy of data C is first written to AZ3. When the subsequent availability of AZ2 rises to a preset threshold, the file system 102 writes a copy of data B into AZ2 according to the data copy stored in AZ1 or AZ3.

It is worth noting that the above four implementation methods are only used as some exemplary illustrations. In practical applications, the file system 102 can also store multiple copies of data in different AZs based on other methods. For example, the file system 102 can be combined In the above implementation manner, multiple AZs with high availability and close physical distances are selected to store multiple copies of data.

In actual application, the distributed database 101 and the file system 102 may include multiple identical AZs, or may not include the same AZ, which is not limited in this embodiment. In a further possible implementation, when the distributed database 101 and the file system 102 can include multiple identical AZs, the file system 102 can track multiple target AZs that store multiple partition copies during the process of storing data copies (The file system 102 also includes the multiple target AZs), so the file system 102 can also store multiple data copies in data nodes under different target AZs. At this time, the file system 102 and the distributed database 102 may store data copies and partition copies in the same AZ based on the same allocation indication information. In this way, when providing data read and write services, the distributed database 101 can read the target data from the local (that is, the AZ where the partition copy is located) data node based on the partition copy, thereby reducing the time delay for reading data and improving The data processing platform 100 feeds back the efficiency of the data to the user. In addition, when the distributed database 101 and the file system 102 do not include the same AZ, the distributed database 101 and the file system 102 can independently execute the stored procedure of the copy.

Moreover, assuming that the number of AZs storing data copies and partition copies is N (N is a positive integer greater than 1), even if up to N-1 AZs crash, the data processing platform 100 can also be based on the partitions stored in the Nth AZ Replicas and data replicas continue to provide users with data read and write services. Moreover, when the remaining N-1 AZs resume operation, the data processing platform 100 can also automatically restore the data copies and partition copies stored in the remaining N-1 AZs based on the partition copies and data copies stored in the Nth AZ. Data, so that there is no need for management personnel to intervene, and automatic processing after fault recovery is realized.

It is worth noting that in this embodiment, the target data and the partition to which the target data belongs are copied and stored across AZs as an example. For other data, the data processing platform 100 can store them in a similar manner as described above to improve data The processing platform 100 provides reliability of data reading and writing services. In practical applications, there may be differences in the importance of different data based on users. For example, for users, data A is more important, while data B is less important (even if data B is lost, it will not affect the The user has a greater influence), therefore, the data processing platform 100 can store the user's data A in the above-mentioned manner, and when storing the user's data B, there is no need to copy and store the data B, or multiple data of the data B Replicas are stored in one AZ.

Further, the data processing platform 100 may also periodically balance the loads of the RS nodes under each AZ. For example, the data processing platform 100 can periodically obtain the number of partition copies stored by the RS nodes under each AZ, so that the RS nodes under different AZs in the distributed database 101 can be loaded according to the number of partition copies in each RS node. Balance, specifically, may be migrating partition copies on some RS nodes to other RS nodes, so as to adjust the number of partition copies stored in different AZs, thereby reducing the difference in the number of partition copies stored in different AZs. For example, assuming that the number of partition copies stored by the RS node under AZ1 is 1, the number of partition copies stored by the RS node under AZ2 is 2, and the number of partition copies stored by the RS node under AZ3 is 3, then the data processing platform 100 A partition copy on the RS node under AZ3 can be migrated to the RS node under AZ1, so that the number of partition copies stored in each AZ is 2.

It should be noted that in this embodiment, the data to be stored includes the target data and the partition to which the target data belongs as an example. In other possible embodiments, the data to be stored may only be the target data. At this time, the data processing platform 100 can store multiple copies of the target data in different AZs through the file system 102, and the partition to which the target data belongs can be stored in a single AZ (one partition data can be saved in this AZ, It is also possible to store multiple copies of the partition at the same time), or it can be stored in multiple AZs. Alternatively, the data to be stored can also be only the partition to which the target data belongs. At this time, the data processing platform 100 can store the data of multiple partitions in different AZs through the distributed data set 101, and the target data can be stored in the file system In a single AZ under the file system 102 (one copy of the target data can be stored in the AZ, or multiple copies of the target data can be stored at the same time), or can be stored in multiple AZs under the file system 102.

The data storage method provided by the embodiment of the present application is introduced above with reference to FIG. 1 to FIG. 4 . Next, the device for executing the above-mentioned data storage and the computing device provided by the embodiment of the present application are introduced with reference to the accompanying drawings.

FIG. 5 is a schematic structural diagram of a data storage device provided by the present application. The data storage device 500 can be applied to a data processing platform (such as the above-mentioned data processing platform 100 etc.), and the data processing platform includes multiple availability zones AZ. Wherein, the data recovery device 500 includes:

An acquisition module 501, configured to acquire multiple copies of the data to be stored;

The storage module 502 is configured to store multiple copies of the data to be stored in different AZs of the data processing platform.

In a possible implementation manner, the data to be stored includes target data and/or the partition to which the target data belongs, and the storage module 502 is configured to:

storing multiple data copies of the target data in different AZs of the data processing platform;

And/or, storing multiple partition copies of the partition to which the target data belongs to different AZs of the data processing platform.

In a possible implementation manner, the data to be stored includes target data and the partition to which the target data belongs, the data processing platform includes a distributed database and a file system, and the distributed database includes multiple availability zones AZ The RS node of the partition server, the file system includes data nodes under multiple AZs, and the storage module 502 is used for:

storing the plurality of partition copies to RS nodes under different AZs in the distributed database;

The multiple data copies are stored in data nodes under different AZs in the file system.

In a possible implementation manner, the storage module 502 is configured to:

Obtain the physical distance between different AZs in the file system;

determining multiple first AZs in the file system according to physical distances between different AZs in the file system, where the physical distances between the multiple first AZs do not exceed a distance threshold;

storing the multiple copies of data to data nodes under the multiple first AZs.

In a possible implementation manner, the storage module 502 is configured to:

Obtain the availability of each AZ in the file system;

According to the degree of availability of each AZ in the file system, the multiple data copies are stored in the data nodes under the multiple first AZs in the file system, and the file system also includes at least one data node under the second AZ For a data node, the availability of the second AZ is lower than the availability of the first AZ.

In a possible implementation manner, the storage module 502 is configured to:

Obtain the availability of each AZ in the file system;

According to the degree of availability of each AZ in the file system, part of the data copies in the multiple data copies are stored in the data nodes under the first AZ in the file system, and the file system also includes at least one second For data nodes under the AZ, the availability of the second AZ is lower than the availability of the first AZ;

When the availability of the at least one second AZ rises to a preset threshold, store other data copies in the plurality of data copies to data nodes under the at least one second AZ.

In a possible implementation manner, the storage module 502 is configured to:

Acquire allocation indication information for the multiple partition copies, where the allocation indication information is used to indicate the proportions of the number of copies stored in different AZs for the multiple partition copies;

According to the allocation indication information, store the multiple partition copies to RS nodes under different AZs in the distributed database.

In a possible implementation manner, the allocation indication information is determined according to the load of the RS nodes under each AZ in the distributed database.

In a possible implementation manner, both the distributed database and the file system include multiple target AZs, and the storage module 502 is configured to:

tracking a plurality of target AZs storing said plurality of partition copies;

The multiple data copies are stored in data nodes under different target AZs.

In a possible implementation manner, after storing the multiple partition copies to RS nodes under different AZs in the distributed database, the apparatus 500 further includes:

The load balancing module 503 is configured to perform load balancing on RS nodes under different AZs in the distributed database, so as to adjust the number of partition copies stored in different AZs.

The data storage device 500 according to the embodiment of the present application may correspond to the implementation of the method described in the embodiment of the present application, and the above-mentioned and other operations and/or functions of the various modules of the data storage device 500 are respectively in order to realize the method embodiment shown in FIG. 3 The corresponding process in , for the sake of brevity, will not be repeated here.

Figure 6 provides a computing device. As shown in Figure 6, the computing device 600 may be, for example, the device used to implement the functions of the data processing platform 100 in the foregoing embodiments, and the computer device 600 may specifically be used to implement the data storage device 500 in the embodiment shown in Figure 5 above function.

The computing device 600 includes a bus 601 , a processor 602 and a memory 603 . The processor 602 and the memory 603 communicate through the bus 601 .

The bus 601 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 6 , but it does not mean that there is only one bus or one type of bus.

The processor 602 may be a central processing unit (central processing unit, CPU), a graphics processing unit (graphics processing unit, GPU), a microprocessor (micro processor, MP) or a digital signal processor (digital signal processor, DSP) etc. Any one or more of them.

The memory 603 may include a volatile memory (volatile memory), such as a random access memory (random access memory, RAM). Memory 603 can also include non-volatile memory (non-volatile memory), such as read-only memory (read-only memory, ROM), flash memory, mechanical hard disk (hard drive drive, HDD) or solid state hard disk (solid state drive) , SSD).

Executable program codes are stored in the memory 603 , and the processor 602 executes the executable program codes to implement the data storage method performed by the aforementioned data processing platform 100 .

The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state hard disk), etc. The computer-readable storage medium includes instructions for instructing a computing device to execute the above data recovery method.

The embodiment of the present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computing device, the processes or functions according to the embodiments of the present application will be generated in whole or in part.

The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g. (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.) to another website site, computer or data center.

The computer program product may be a software installation package which can be downloaded and executed on a computing device if any of the aforementioned object recognition methods are required.

The description of the process or structure corresponding to each of the above drawings has its own emphasis. For the part that is not described in detail in a certain process or structure, you can refer to the relevant description of other processes or structures.

Claims

A data storage method, characterized in that the method is applied to a data processing platform, and the data processing platform includes multiple availability zones AZ, and the method includes:

Obtain multiple copies of the data to be stored;

storing multiple copies of the data to be stored in different AZs of the data processing platform.
The method according to claim 1, wherein the data to be stored includes target data and/or the partition to which the target data belongs, and storing multiple copies of the data to be stored to the data processing platform in different AZs, including:

storing multiple data copies of the target data in different AZs of the data processing platform;

And/or, storing multiple partition copies of the partition to which the target data belongs to different AZs of the data processing platform.
The method according to claim 2, wherein the data to be stored includes target data and the partition to which the target data belongs, the data processing platform includes a distributed database and a file system, and the distributed database includes multiple The partition server RS node under the availability zone AZ, the file system includes data nodes under multiple AZs, and storing multiple copies of the data to be stored in different AZs of the data processing platform includes:

storing the plurality of partition copies to RS nodes under different AZs in the distributed database;

The multiple data copies are stored in data nodes under different AZs in the file system.
The method according to claim 3, wherein the storing the multiple data copies in data nodes under different AZs in the file system comprises:

Obtain the physical distance between different AZs in the file system;

determining multiple first AZs in the file system according to physical distances between different AZs in the file system, where the physical distances between the multiple first AZs do not exceed a distance threshold;

storing the multiple copies of data to data nodes under the multiple first AZs.
The method according to claim 3, wherein the storing the multiple data copies in data nodes under different AZs in the file system comprises:

Obtain the availability of each AZ in the file system;

According to the degree of availability of each AZ in the file system, storing the multiple data copies to data nodes under multiple first AZs in the file system, the file system also includes at least one second AZ, The degree of availability of the second AZ is lower than the degree of availability of the first AZ.
The method according to claim 3, wherein the storing the multiple data copies in data nodes under different AZs in the file system comprises:

Obtain the availability of each AZ in the file system;

According to the degree of availability of each AZ in the file system, part of the data copies in the multiple data copies are stored in the data nodes under the first AZ in the file system, and the file system also includes at least one second AZ, the availability of the second AZ is lower than the availability of the first AZ;

When the availability of the at least one second AZ rises to a preset threshold, store other data copies in the plurality of data copies to data nodes under the at least one second AZ.
The method according to any one of claims 1 to 6, wherein the storing the multiple partition copies to RS nodes under different AZs in the distributed database comprises:

Acquire allocation indication information for the multiple partition copies, where the allocation indication information is used to indicate the proportions of the number of copies stored in different AZs for the multiple partition copies;

According to the allocation indication information, store the multiple partition copies to RS nodes under different AZs in the distributed database.
The method according to claim 7, wherein the allocation indication information is determined according to the load of RS nodes under each AZ in the distributed database.
The method according to claim 3, wherein the distributed database and the file system both include multiple target AZs, and the multiple data copies are stored in different AZs in the file system Data nodes, including:

tracking a plurality of target AZs storing said plurality of partition copies;

The multiple data copies are stored in data nodes under different target AZs.
The method according to any one of claims 3 to 9, wherein after storing the plurality of partition copies to RS nodes under different AZs in the distributed database, the method further comprises:

Perform load balancing on the RS nodes under different AZs in the distributed database, so as to adjust the number of partition copies stored in different AZs.
A data storage device, characterized in that the device is applied to a data processing platform, the data processing platform includes multiple availability zones AZ, and the device includes:

An acquisition module, configured to acquire multiple copies of the data to be stored;

A storage module, configured to store multiple copies of the data to be stored in different AZs of the data processing platform.
The device according to claim 11, wherein the data to be stored includes target data and/or the partition to which the target data belongs, and the storage module is configured to:

storing multiple data copies of the target data in different AZs of the data processing platform;

And/or, storing multiple partition copies of the partition to which the target data belongs to different AZs of the data processing platform.
The device according to claim 12, wherein the data to be stored includes target data and the partition to which the target data belongs, the data processing platform includes a distributed database and a file system, and the distributed database includes multiple The partition server RS node under the availability zone AZ, the file system includes data nodes under multiple AZs, and the storage module is used for:

storing the plurality of partition copies to RS nodes under different AZs in the distributed database;

The multiple data copies are stored in data nodes under different AZs in the file system.
The device according to claim 13, wherein the storage module is used for:

Obtain the physical distance between different AZs in the file system;

determining multiple first AZs in the file system according to physical distances between different AZs in the file system, where the physical distances between the multiple first AZs do not exceed a distance threshold;

storing the multiple copies of data to data nodes under the multiple first AZs.
The device according to claim 13, wherein the storage module is used for:

Obtain the availability of each AZ in the file system;

According to the degree of availability of each AZ in the file system, storing the multiple copies of data to data nodes under multiple first AZs in the file system, the file system also includes at least one second AZ, so The degree of availability of the second AZ is lower than the degree of availability of the first AZ.
The device according to claim 13, wherein the storage module is used for:

Obtain the availability of each AZ in the file system;

According to the degree of availability of each AZ in the file system, part of the data copies in the multiple data copies are stored in the data nodes under the first AZ in the file system, and the file system also includes at least one second AZ, the availability of the second AZ is lower than the availability of the first AZ;

When the availability of the at least one second AZ rises to a preset threshold, store other data copies in the plurality of data copies to data nodes under the at least one second AZ.
The device according to any one of claims 11 to 16, wherein the storage module is used for:

Acquire allocation indication information for the multiple partition copies, where the allocation indication information is used to indicate the proportions of the number of copies stored in different AZs for the multiple partition copies;

According to the allocation indication information, store the multiple partition copies to RS nodes under different AZs in the distributed database.
The device according to claim 17, wherein the allocation indication information is determined according to the load of the RS nodes under each AZ in the distributed database.
The device according to claim 13, wherein the distributed database and the file system both include multiple target AZs, and the storage module is configured to:

tracking a plurality of target AZs storing said plurality of partition copies;

The multiple data copies are stored in data nodes under different target AZs.
The device according to any one of claims 13 to 19, wherein after storing the multiple partition copies to RS nodes under different AZs in the distributed database, the device further comprises:

A load balancing module, configured to perform load balancing on RS nodes under different AZs in the distributed database, so as to adjust the number of partition copies stored in different AZs.
A computing device, characterized in that it includes a processor and a memory;

The processor is configured to execute instructions stored in the memory, so that the computing device executes the method according to any one of claims 1 to 10.
A computer-readable storage medium, characterized by comprising instructions, which, when run on a computing device, cause the computing device to execute the method according to any one of claims 1 to 10.
A computer program product comprising instructions which, when run on a computing device, cause the computing device to perform the method as claimed in any one of claims 1 to 10.