CN116431594A

CN116431594A - Data storage method and device and computer readable storage medium

Info

Publication number: CN116431594A
Application number: CN202210003129.2A
Authority: CN
Inventors: 孙大鹏
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Suzhou Software Technology Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Suzhou Software Technology Co Ltd
Priority date: 2022-01-04
Filing date: 2022-01-04
Publication date: 2023-07-14

Abstract

The embodiment of the invention discloses a data storage method, a data storage device and a storage medium, and a file uploading request sent by a terminal is received; according to the file uploading request, determining the storage position of the file by using a self-defining method and sending the file to the terminal; receiving a file access request sent by a terminal; determining a target data center in preset data centers according to the file access request; presetting a storage position of a file contained in a data center; and acquiring the stored file according to the storage position through the target data center, and sending the file to the terminal. In the above scheme, on one hand, a storage position of a file is selected in a preset data center by a self-defining method; on the other hand, at the time of file reading, a target data center is determined among preset data centers including a plurality of storage locations, and then file reading is performed from the target data center. The whole scheme is integrated, so that resources are saved, the complexity of the system is simplified, and the possibility of faults is reduced.

Description

Data storage method and device and computer readable storage medium

Technical Field

The present invention relates to the field of big data technologies, and in particular, to a data storage method and apparatus, and a computer readable storage medium.

Background

With the development of big data, the dependence degree of each industry on the data is higher. In order to ensure the accuracy and reliability of data, the necessity of data disaster recovery is particularly important. The disaster recovery of the HBase is generally referred to as data center-level disaster recovery, generally, a machine room cluster server disaster recovery technology is adopted, two sets of HBase clusters are respectively deployed in different data centers, and then data are written into two clusters respectively through a certain technical means or written into a certain main cluster first and then copied into a backup cluster, so that the final consistency of the data of the two clusters is ensured. Therefore, when a certain data center fails, the data service is switched to be provided by the backup cluster through technical means such as failover, so that the data center disaster recovery of the HBase cluster is ensured.

However, in the data storage, only the primary cluster provides service, and the backup cluster only performs data synchronous backup in a normal state, but does not provide service. In order to ensure that the backup clusters can synchronize the data of the main clusters well and ensure the reliability of the data, the resource allocation of the backup clusters cannot have larger difference with the main clusters, so that the computing resources of the backup clusters are completely idle at ordinary times. In the dual-active disaster recovery scheme, two clusters are the main and the standby, and provide services at the same time, so that the computing resources of the two clusters are fully utilized, but in order to ensure that uniform and consistent services are provided for the service, a certain technical means is needed to ensure that only one cluster provides services at the same time, otherwise, two clusters repeatedly compute, and the computing resources are wasted. hdfs typically records three copies at different storage locations, which means that for a dual HBase cluster of a master/slave/dual active scheme, one copy of data will typically be stored 6 copies, which is a significant waste of storage resources. On the other hand, in the main/standby/dual-active/dual-cluster scheme, only the main cluster of a certain service provides service in a normal state, and the service is switched to be provided by the standby cluster through a certain technical means only when the main cluster fails. There is an inevitable need to additionally design cluster health monitoring modules and fail-over modules. The complexity, implementation difficulty and fault probability of the system are all relatively large.

Disclosure of Invention

The embodiment of the invention provides a data storage method and device and a computer readable storage medium, which can save resources, simplify the complexity of a system and reduce the possibility of faults.

The technical scheme of the invention is realized as follows:

the embodiment of the invention provides a data storage method which is applied to a server and comprises the following steps:

receiving a file uploading request sent by a terminal;

determining a storage position of the file by using a self-defining method according to the file uploading request and sending the file to a terminal;

receiving a file access request sent by the terminal;

determining a target data center in a preset data center according to the file access request; the preset data center comprises a storage position of the file;

and acquiring the stored file according to the storage position through the target data center, and sending the file to the terminal.

In the above scheme, the preset data center comprises a first data center and a second data center; the first data center and the second data center each include a plurality of different racks; the plurality of different racks respectively comprise a plurality of different data nodes; taking the distance value between the data nodes of different racks of the first data center as a first distance value; the distance value between the data nodes of different racks of the second data center is a second distance value;

The step of determining the storage position of the file and sending the storage position to the terminal by using a self-defining method according to the file uploading request comprises the following steps:

acquiring data nodes of the first data center and data nodes of the second data center;

and according to the first distance value and the second distance value, combining the data nodes of the first data center and the data nodes of the second data center, determining the storage position of the file and sending the storage position to the terminal.

In the above solution, the determining, according to the first distance value and the second distance value, the storage location of the file in combination with the node of the first data center and the node of the second data center, and sending the storage location to the terminal includes:

acquiring a first data node;

according to the first data node, combining the first distance value and the second distance value to respectively determine a second data node, a third data node and a fourth data node;

and determining the storage position of the file according to the first data node, the second data node, the third data node and the fourth data node, and sending the file to the terminal.

In the above scheme, the acquiring the first data node includes:

Traversing a plurality of data nodes in a cluster, and randomly acquiring the first data node from the plurality of data nodes if the terminal is not in the cluster; the cluster comprises the plurality of data nodes;

and if the terminal is in the cluster, the data node where the terminal is located is the first data node.

In the above scheme, the sum of distances between any two data nodes among the first data node, the second data node, the third data node and the fourth data node is a preset value;

the determining, according to the first data node, the second data node, the third data node, and the fourth data node by combining the first distance value and the second distance value, includes:

if the first data node is a node of the first data center, determining the second data node according to the first data node and the first distance value;

determining the third data node according to the second data node and the second data center; the third data node is any node of the second data center;

determining a fourth data node according to the third data node and the second distance value;

if the first data node is a node of the second data center, determining the second data node according to the first data node and the second distance value;

determining the third data node according to the second data node and the first data center; the third data node is any node of the first data center;

and determining the fourth data node according to the third data node and the first distance value.

In the above scheme, the determining, according to the file access request, the target data center in the preset data center includes:

comparing the distances between the first data center and the second data center and the terminal respectively; determining a comparison result;

determining a data center closest to the data center according to the comparison result, wherein the data center closest to the data center is a target data center;

In the above solution, the obtaining, by the target data center, the stored file according to the storage location, and sending the file to the terminal includes:

if the first data center is the target data center and the first data center is detected to be not faulty, acquiring the file according to the storage position in the first data center and sending the file to the terminal;

and if the first data center is the target data center and the first data center is detected to be faulty, acquiring the file according to the storage position in the second data center and sending the file to the terminal.

In the above solution, if the first data center is the target data center and it is detected that the first data center fails, the acquiring, in the second data center, the file according to the storage location and sending the file to the terminal includes:

restarting related services in the second data center when the first data center is detected to be faulty through the deployed monitoring and identifying service layer;

and in a second data center after the related service is started, acquiring the file according to the storage position and sending the file to the terminal.

The embodiment of the invention provides a data storage method, which is applied to a terminal and comprises the following steps:

sending a file uploading request to a server;

receiving a file storage position sent by the server according to the file uploading request;

and according to the received storage positions, sequentially sending the files to the storage positions for storage. In the above solution, after the files are sequentially sent to the storage locations according to the storage locations for storage, the method further includes:

if the file stored by the terminal is damaged, sending a file access request to the server;

and receiving the file sent by the server through the storage position for substitution based on the file access request.

The embodiment of the invention provides a data storage device, which comprises a first receiving unit, a first determining unit and a first transmitting unit; wherein,,

the first receiving unit is used for receiving a file uploading request sent by the terminal;

the first sending unit is used for determining the storage position of the file by using a self-defining method according to the file uploading request and sending the file to the terminal;

The first receiving unit is also used for receiving a file access request sent by the terminal;

a first determining unit, configured to determine a target data center in a preset data center according to the file access request; the preset data center comprises a storage position of the file;

the first sending unit is further configured to obtain, through the target data center, the stored file according to the storage location, and send the file to the terminal.

The embodiment of the invention provides a data storage device, which comprises a second sending unit and a second receiving unit; wherein,,

the second sending unit is used for sending a file uploading request to the server;

the second receiving unit is used for receiving the file storage position sent by the server according to the file uploading request;

and the second sending unit is also used for sending the files to the storage positions for storage according to the received storage positions in sequence.

The embodiment of the invention provides a server, which comprises:

a first memory for storing executable instructions;

and the first processor is used for executing the executable instructions stored in the first memory, and when the executable instructions are executed, the first processor executes the data storage method applied to the server side.

The embodiment of the invention provides a terminal, which comprises:

a second memory for storing executable instructions;

and a second processor for executing the executable instructions stored in the second memory, the second processor executing the data storage method applied to the terminal when the executable instructions are executed.

Embodiments of the present invention provide a storage medium storing executable instructions that, when executed, are configured to cause a first processor to perform a data storage method according to embodiments of the present invention.

Embodiments of the present invention provide a storage medium storing executable instructions that, when executed, are configured to cause a second processor to perform a data storage method according to embodiments of the present invention.

The embodiment of the invention provides a data storage method and device and a computer readable storage medium, wherein the method comprises the following steps: receiving a file uploading request sent by a terminal; determining a storage position of the file by using a self-defining method according to the file uploading request and sending the file to a terminal; receiving a file access request sent by the terminal; determining a target data center in a preset data center according to the file access request; the preset data center comprises a storage position of the file; and acquiring the stored file according to the storage position through the target data center, and sending the file to the terminal. By adopting the scheme, on one hand, the storage position of the file is selected from a preset data center by a self-defining method; on the other hand, at the time of file reading, a target data center is determined among preset data centers including a plurality of storage locations, and then file reading is performed from the target data center. The whole scheme is integrated, so that resources are saved, the complexity of the system is simplified, and the possibility of faults is reduced.

Drawings

Fig. 1 is a flowchart illustrating a method for data storage applied to a server according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating a method for storing data applied to a server according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an adjacency matrix effect of a storage data center node according to an embodiment of the present invention;

fig. 4 is a flowchart illustrating a method for storing data applied to a server according to an embodiment of the present invention;

fig. 5 is a flowchart illustrating a method for storing data applied to a server according to an embodiment of the present invention;

fig. 6 is a flowchart of a method for storing data applied to a server according to an embodiment of the present invention;

fig. 7 is a flowchart illustrating a method for data storage applied to a server according to an embodiment of the present invention;

FIG. 8 is a block diagram of a metadata management module according to an embodiment of the present invention;

fig. 9 is a block diagram of an HBase disaster recovery system across a data center according to an embodiment of the present invention;

fig. 10 is a flowchart of a method for data storage applied to a terminal according to an embodiment of the present invention;

FIG. 11 is a flowchart of a method for data storage according to an embodiment of the present invention;

FIG. 12 is a schematic diagram of a data storage device according to an embodiment of the present invention;

FIG. 13 is a schematic diagram of another data storage device according to an embodiment of the present invention;

fig. 14 is a schematic structural diagram of a server according to an embodiment of the present invention;

fig. 15 is a schematic structural diagram of a terminal according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without making any inventive effort are within the scope of the present invention.

In order to better understand the aspects of the present invention, the present invention will be described in further detail with reference to the accompanying drawings and detailed description. Fig. 1 is a flowchart of a method for storing data applied to a server according to an embodiment of the present invention, and will be described with reference to the steps shown in fig. 1.

S101, receiving a file uploading request sent by a terminal.

In the embodiment of the invention, a client wants to store a file, sends a file uploading request to a server, and after the server acquires the file uploading request sent by a terminal, the server performs subsequent operations.

S102, determining a storage position of the file by using a self-defining method according to the file uploading request, and sending the file to the terminal.

In the embodiment of the invention, after the server acquires the file uploading request sent by the terminal, a specific storage position of the file is determined by using a self-defining method, and the position is sent to the client.

In the embodiment of the invention, the data of the HBase is stored on the hdfs, and the hdfs itself adopts a multi-copy mechanism, usually three copies, in order to ensure the reliability of the data, i.e. for one data, the hdfs usually records three copies in different storage positions. This means that for a dual HBase cluster of the active/dual active scheme, a copy of data will typically be stored for 6 copies, which is a significant waste of storage resources. In the invention, only one HBase cluster is deployed by using a self-defining method, and the cross-data center disaster recovery of the storage layer is realized by modifying the copy placement strategy of the hdfs. For one piece of data, only 4 copies need to be saved, and finally the determined 4 positions are sent to the client.

S103, receiving a file access request sent by the terminal.

In the embodiment of the invention, a server receives a file access request sent by a terminal.

In the embodiment of the invention, after the terminal stores the file, the stored file is read again, a file access request is sent to the server, and the server receives the file and then performs subsequent operations.

S104, determining a target data center in preset data centers according to the file access request; the preset data center comprises a storage position of the file.

In the embodiment of the invention, after the client stores the file according to the position sent by the server, if the client wants to acquire the file again, the client sends a file access request to the server, and the server determines a target data center in a plurality of data centers containing the file storage position according to the file access request.

In the embodiment of the invention, the preset data centers all contain the storage positions of the files, and then the target data centers meeting the requirements are found in the preset data centers according to the preset conditions.

S105, acquiring the stored file according to the storage position through the target data center, and sending the file to the terminal.

In the embodiment of the invention, after the target data center is found, the stored file is acquired according to the stored position and is sent to the terminal.

It can be understood that in the embodiment of the present invention, the terminal sends a file upload request to the server, and the server selects a storage location of the file by using a custom method. After the storage is completed, the terminal sends a file access request, and the server acquires the stored file according to the file storage position and sends the file to the terminal after finding the target data center. According to the scheme, the copy placement strategy of hdfs is improved, the hmaster and the regionserver of the HBase cluster are deployed in a containerization mode, resources are saved, the complexity of the system is simplified, and the possibility of faults is reduced.

In some embodiments of the present invention, referring to fig. 2, fig. 2 is a second flowchart of a method for data storage applied to a server according to an embodiment of the present invention, S102 shown in fig. 2 may be implemented by S1021 to S1022, and the steps shown in fig. 2 will be described.

S1021, acquiring a data node of a first data center and a data node of a second data center; the plurality of different racks respectively comprise a plurality of different data nodes; taking the distance value between the data nodes of different racks of the first data center as a first distance value; and taking the distance value between the data nodes of different racks of the second data center as a second distance value.

In some embodiments of the present invention, a plurality of data nodes are partitioned to determine data nodes contained in a first data center and data nodes contained in a second data center.

In some embodiments of the present invention, the first data center and the second data center each comprise a plurality of racks, and the nodes respectively comprised by the first data center and the second data center may also be separated into different racks.

For example, if there are 12 data nodes, A, B, C … L each, where A-F belongs to a first data center, A-C belongs to the same rack of the first data center, and D-F belongs to another rack of the first data center. G-L belongs to the second data center, G-I belongs to the same rack of the second data center, and I-L belongs to another rack of the second data center.

S1022, according to the first distance value and the second distance value, combining the data node of the first data center and the data node of the second data center, determining the storage position of the file, and sending the file to the terminal.

In some embodiments of the invention, the distance value between nodes that belong to the same first data center but are different racks in the first data center is set to a first distance value.

As shown in fig. 3, fig. 3 is a schematic view of an adjacency matrix effect of nodes of a storage data center, wherein values in the matrix are distance values between nodes of the same data center and the same rack respectively; distance values between nodes of the same data center, different racks, and distance values between nodes of different data centers. For example, the node a and the node D, E, F are each a node of the first data center, but are nodes of different racks, so the distance between the node a and the node D, E, F is set to a first distance value, which may be 2. And vice versa for other data nodes.

In some embodiments of the invention, the distance between nodes belonging to the same second data center, but different racks in the second center, is set to a second distance value.

Illustratively, as shown in fig. 3, the node G and the node J, K, L are each a node of a second data center, but are nodes of different racks, so the distance from the node G to the node J, K, L, respectively, is set to a second distance value, which may be 3. And vice versa for other data nodes.

In some embodiments of the present invention, after the first distance value and the second distance value are obtained, a storage location of the file is determined in combination with the node of the first data center and the node of the second data center, and the determined location is sent to the client.

It will be appreciated that in some embodiments of the present invention, the concept of a data center is added to hdfs, such that when hdfs stores data, only a small number of copies are needed to ensure consistency and reliability of both data center data.

In some embodiments of the present invention, referring to fig. 4, fig. 4 is a flowchart illustrating a third method for data storage applied to a server according to an embodiment of the present invention, S1024 shown in fig. 4 may be implemented through S10241 to S10243, and the steps shown in fig. 4 will be described.

S10241, acquiring a first data node.

In some embodiments of the present invention, a first data node is first acquired, and subsequent operations are performed according to the first data node in combination with a first distance value and a second distance value.

S10242, respectively determining a second data node, a third data node and a fourth data node according to the first data node and combining the first distance value and the second distance value;

in some embodiments of the present invention, after the first data node is acquired, the second data node, the third data node, and the fourth data node are respectively determined in combination with the first distance value and the second distance value.

S10243, determining the storage position of the file according to the first data node, the second data node, the third data node and the fourth data node, and sending the file to the terminal.

In some embodiments of the present invention, after the storage location of the file is obtained according to the obtained four data nodes, the location information is returned to the terminal.

For example, if the first data node is located on a specific rack of the first data center, the characterization file is stored on the specific rack of the first data center, and after a specific storage location of the file is acquired, the location is returned to the terminal.

It can be appreciated that in some embodiments of the present invention, the four data nodes selected are nodes on different data centers and different racks, respectively, and such a placement policy ensures that different data centers have copies of a certain data block, and a certain fault tolerance mechanism exists in any data center.

In some embodiments of the present invention, S10241 can be implemented by S201 and S202, and a specific description is given in connection with the steps.

S201, traversing a plurality of data nodes in a cluster, and randomly acquiring a first data node from preset data nodes if a terminal is not in the cluster; the cluster includes a plurality of data nodes.

In some embodiments of the present invention, a plurality of data nodes in a cluster are traversed, and it is determined that a terminal is not in the cluster, and if the terminal is in the cluster at this time, the data node in which the terminal is located is directly selected as a first data node.

In some embodiments of the present invention, if D1 and D2 are two data centers, R1, R2, R3 are three racks of D1, there are three nodes on R1, N2, and N3, assuming that N1, N2, and N3 are three nodes of one cluster, if the terminal is on N1 at this time, it indicates that the terminal is within the cluster, and if the terminal is on R1 at this time, it indicates that the terminal is not within the cluster. In the invention, only one cluster is deployed, and when the terminal is not in the cluster, the first data node is randomly acquired in a plurality of data nodes.

S202, if the terminal is in the cluster, the data node where the terminal is located is a first data node.

In some embodiments of the present invention, when the terminal is in the cluster, the data node where the terminal is located is the first data node.

In some embodiments of the present invention, referring to fig. 5, fig. 5 is a flowchart illustrating a method for data storage applied to a server according to an embodiment of the present invention, S10242 shown in fig. 5 may be implemented by S301 to S303, and the steps shown in fig. 5 will be described.

S301, if the first data node is a node of the first data center, determining a second data node according to the first data node and the first distance value.

In some embodiments of the present invention, if the first data node is a node of the first data center, the second data node is determined according to the first data node and the first distance value.

In some embodiments of the present invention, if the first data node is a node of the first data center, a satisfactory node is determined in the first data center according to S201 and S202, and then a second data node is determined according to the first data node and the first distance value. As shown in fig. 3, as the first data node is node a in the first data center, since the first distance value is a distance value between data nodes of the first data center and different racks, the second node may select among nodes D, E, F in the first data center.

S302, determining a third data node according to a second data node and a second data center; the third node is any node of the second data center.

In some embodiments of the present invention, after the second data node is obtained, a third data node is determined according to the second data node and the second data center, where the third data node is any node in the second data center.

For example, if the second data node is set to D according to S301, as shown in fig. 3, the distance from the node D to any node of the second data center is 3, so any node is selected as the third node in the second data center.

S303, determining a fourth data node according to the third data node and the second distance value.

In some embodiments of the present invention, after the third data node is acquired, a fourth data node is determined based on the third data node and the second distance value.

In some embodiments of the present invention, because the second distance value is a distance value between data nodes of a second data center, different racks, the fourth data node determined from the third data node and the second distance value is a node on the same data center, different racks.

In some embodiments of the present invention, the sum of distances between any two data nodes of the finally selected first data node, second data node, third data node and fourth data node is a preset value.

For example, if the first data node is a, the second data node is D, the third data node is G, the fourth data node is J, and the preset value may be 16, AD (2) +ag (3) +aj (3) +dg (3) +dj (3) +gj (2) =16.

It can be appreciated that in some embodiments of the present invention, the four selected data nodes are nodes on different data centers and different racks, respectively, and by using the method to determine the storage location of the file, the reliability and fault tolerance of file storage are increased.

In some embodiments of the present invention, referring to fig. 6, fig. 6 is a flowchart of a method for data storage applied to a server according to an embodiment of the present invention, S10242 shown in fig. 6 may be implemented through S401 to S403, and the steps shown in fig. 6 will be described.

S401, if the first data node is a node of the second data center, determining the second data node according to the first data node and the second distance value.

In some embodiments of the present invention, if the first data node is a node of the second data center, the second data node is determined according to the first data node and the second distance value.

In some embodiments of the present invention, if the first data node is a node of the second data center, a satisfactory node is determined in the second data center according to S201 and S202, and then the second data node is determined according to the first data node and the second distance value. As shown in fig. 3, the second data node may be selected among the nodes J, K, L of the second data center because the second distance value is a distance value between the data nodes of the second data center and the different racks, as the first data node is the node G in the second data center.

S402, determining a third data node according to the second data node and the first data center; the third data node is any node of the second data center.

In some embodiments of the present invention, after the second data node is obtained, a third data node is determined according to the second data node and the first data center, where the third data node is any node in the second data center.

For example, if the second data node is set to J according to S401, as shown in fig. 3, the distance between the node J and any node of the first data center is 3, so any node is selected as the third node in the first data center.

S403, determining a fourth data node according to the third data node and the first distance value.

In some embodiments of the invention, after the third node is acquired, a fourth data node is determined from the third node and the first distance value.

In some embodiments of the present invention, because the first distance value is a distance between data nodes of a second data center, different racks, the fourth data node determined from the third data node and the first distance value is a node on the same data center, different racks.

In some embodiments of the present invention, the sum of distances between any two data nodes of the finally selected first, second, third and fourth data nodes is a preset value.

For example, if the first data node is G, the second data node is J, the third data node is D, the fourth data node is a, and the preset value may be 16, GJ (2) +gd (3) +ga (3) +jd (3) +ja (3) +da (2) =16.

In some embodiments of the present invention, S103 may be implemented through S1031 to S1032, which will be specifically described in connection with the steps.

S1031, comparing the distances between the first data center and the second data center and the terminal respectively; and determining a comparison result.

In some embodiments of the present invention, the distance from the first data center to the terminal and the distance from the second data center to the terminal are obtained respectively, and the two distances are compared respectively to determine the comparison result.

In some embodiments of the present invention, the storage location of the file is acquired and then sent to the terminal, and the terminal uploads the file after acquiring the location. After uploading the file, the uploaded file is read, the first data center and the second data center for storing the file are compared, a comparison result is determined, and the file is acquired according to the comparison result.

S1032, determining the nearest data center according to the comparison result; the closest data center is the target data center.

In some embodiments of the present invention, after the comparison result is obtained, the nearest data center is determined according to the comparison result.

In some embodiments of the invention, the nearest data center is selected following the nearest neighbor principle after comparing the first data center with the second data center.

In some embodiments of the present invention, S1033 may be implemented through S10331 to S10332, and the specific explanation is provided in connection with the steps.

S10331, if the first data center is the target data center and failure of the first data center is detected, acquiring a file according to the storage position in the first data center and sending the file to the terminal.

In some embodiments of the present invention, if the first data center is a target data center, it is determined whether the data center fails, and if the data center does not fail, the file is obtained and sent to the terminal after the data center finds the storage location of the file.

S10332, if the first data center is the target data center and the first data center is detected to be faulty, acquiring a file according to the storage position in the second data center and sending the file to the terminal.

In some embodiments of the present invention, if the first data center is a target data center and a failure occurs, the file is acquired and sent to the terminal after the second data center is in accordance with the storage location of the file.

It will be appreciated that in some embodiments of the present invention, nodes within a kubernetes cluster across data centers are labeled with different availability zones, and based on node affinity, the hmaster/regionserver is prioritized over the data center closest to the terminal, and if the closest data center fails, this constraint is ignored and the hmaster/regionserver service is restarted over another data center. This results in a saving of resources for the computing layer.

In some embodiments of the present invention, if the second data center is the closest data center, the operation results are the same as S10331 and S10332.

In some embodiments of the present invention, referring to fig. 7, fig. 7 is a flowchart illustrating a method for data storage applied to a server according to an embodiment of the present invention, S10332 shown in fig. 7 may be implemented through S501 to S502, and the steps shown in fig. 7 will be described.

S501, through a deployed service layer identified by monitoring, if the first data center is detected to be faulty, the related service is started in the second data center.

In some embodiments of the present invention, if a failure is detected in the data center closest to the service layer identified by the deployed monitoring, the relevant service is started in the second data center.

In some embodiments of the invention, a kubernetes cluster across data centers is employed for containerized deployment. Considering that the terminal accesses the HBase cluster of the data center which is closer in network topology as far as possible, marking different available areas for nodes in the kubernetes cluster crossing the data center, enabling the hmaster/regionserver to preferentially play on the first data center according to the node affinity, and restarting the hmaster/regionserver service on the second data center if the first data center breaks down, and ignoring the limiting condition.

In some embodiments of the present invention, in order to facilitate a terminal to access an HBase cluster, a unified service is adopted for service exposure on web UIs of a containerized deployment hmaster and a regional server, without causing a change in cluster connection information due to a change in a hmaster/regional server container. That is, a layer of monitoring and identifying service is deployed on the containers of the hmaster and the region server, the service can monitor the change of the container ip of the hmaster/region server in real time, the container ip is not directly exposed, the container ip is bound with a domain name of a fixed service exposure, and the terminal only accesses through the fixed domain name. When the service request reaches the monitoring service layer, the service request is parsed and forwarded to the corresponding hmaster/regionserver. Therefore, even if the main data center fails, the HBase cluster related container is restarted on the node of the other data center, the connection information of the HBase cluster is not changed, and the terminal does not feel.

In some embodiments of the present invention, for HBase clusters, zookeeper storing metadata is also a critical part, and disaster recovery is also required to be considered to be high available, for zookeeper, kubernetes clusters crossing data centers are also adopted to perform containerized deployment, and data of zookeeper is stored externally, so that even if zookeeper container is restarted in another data center, consistency of zookeeper data can be ensured. The structure of the metadata management module zookeeper is shown in fig. 8, and the zookeeper is mainly used for solving some data management problems frequently encountered in the distributed application, such as: unified naming service, state synchronization service, cluster management, management of distributed application configuration items, etc., zone a and Zone B are respectively different data centers, node is a data node, ceph is a unified, distributed file system designed for excellent performance, reliability and expandability, object storage, block device storage and file system service can be provided, kubernetes is a container arrangement engine of Google open source, which supports automated deployment, large-scale scalability and application containerization management. When an application is deployed in a production environment, multiple instances of the application are typically deployed to load balance application requests. In Kubernetes, multiple containers may be created, one application instance running in each container, and then management, discovery, and access to the set of application instances is implemented through a built-in load balancing policy, where no complex manual configuration and processing by operation and maintenance personnel is required for these details.

S502, acquiring the file according to the storage position of the file in a second data center after the related service is started, and sending the file to the terminal.

In some embodiments of the present invention, after detecting that the latest data center fails, the relevant service is restarted on a node of the second data center, then a storage location of the file is found in the second data center, and the file is acquired according to the location and then sent to the terminal.

In some embodiments of the present invention, the related service refers to hmaster/region server service, the structure diagram of the whole scheme is shown in fig. 9, fig. 9 is a structure diagram of an HBase disaster recovery system across data centers provided in the embodiments of the present invention, a storage layer is adopted, an hdfs cluster across data centers is adopted, data adopts a 4-copy placement strategy, for example, red data block1, is respectively placed on different nodes of different racks of different zones (for example,

red nodes

1, 2, 3 and 4 in the figure) to implement fault tolerance across data center levels; and in a normal state, all containers for deploying services are in a zone A, when the zone A fails, the related container services are restarted in a zone B, so that data center level fault tolerance is also realized, rack1 and rack2 are racks, and HMaster is a central node in a Hbase master/slave cluster architecture. Typically, one HBase cluster has multiple HMaster nodes, one of which is an Active Master, the rest is a Backup Master, and only one HMaster main server program is running at any time by the HBase, and the HMaster allocates regions to the region servers, coordinates the loads of the region servers and maintains the state of the cluster. The Hmaster does not provide data services to the outside, but the region server takes charge of read-write requests and operations of all regions. Because hmaster only maintains metadata of tables and regions and does not participate in the data input/output process, hmaster failure only causes all metadata to be unable to be modified, but data reading/writing of tables can be performed normally.

It can be appreciated that in some embodiments of the present invention, only one set of HBase computing resources is deployed in a containerized manner, and when a certain data center fails, the HBase computing resource container is restarted in another data center, so that the computing resources are saved and meanwhile, the service continuity is ensured.

In the embodiment of the present invention, referring to fig. 10, fig. 10 is a flowchart of a method applied to data storage of a terminal according to an embodiment of the present invention, and will be specifically described with reference to the content shown in fig. 10.

S601, sending a file uploading request to a server.

In the embodiment of the invention, when the terminal wants to store the file, the terminal sends a file uploading request to the server.

S602, receiving a file storage position sent by a server according to a file uploading request.

In the embodiment of the invention, the server determines the storage position of the file according to the file uploading request sent by the terminal, and sends the file to the terminal, and then the terminal receives the file storage position sent by the server.

And S603, according to the received storage positions, sequentially sending the files to the storage positions for storage.

In an embodiment of the present invention, files are sequentially transmitted to a server according to received file storage locations, so that the server stores the files.

In the embodiment of the invention, 4 copies are stored together, and when the copy 1 is uploaded to the corresponding position, the copy 2 is uploaded.

In some embodiments of the present invention, S603 further includes S701, and S701 may be implemented through S7011 to S7012, which are specifically described in connection with the steps.

S7011, if the file stored in the terminal is damaged, a file access request is sent to the server.

In the embodiment of the invention, after the files are stored, if the files stored by the terminal are damaged, the server is required to send a file access request.

S7012, based on the file access request, the receiving server replaces the file transmitted through the storage location.

In the embodiment of the invention, after a file access request is sent to a server, the server acquires the stored file and then sends the file to a terminal, and the terminal receives the file sent by the server.

In the embodiment of the invention, the terminal picks a (nearby principle, then randomly) server to request to read the data.

It can be understood that in the embodiment of the invention, the terminal performs file storage according to the number of the stored files sent by the server and different from the number of the existing stored files by sending the file uploading request to the server, and utilizes the sent file access request, so that the fault tolerance of file storage is improved, and storage and resources are saved.

An alternative flow chart of a method for storing data according to an embodiment of the present invention is shown in fig. 11.

S1, a terminal sends a file uploading request to a server.

S2, the server receives a file uploading request sent by the terminal.

And S3, the server determines the storage position of the file by using a self-defining method according to the received file uploading request and sends the file to the terminal.

And S4, after the terminal sends the file uploading request, receiving the file storage position sent by the server.

And S5, the terminal sequentially sends the files to the server according to the file storage positions, so that the server stores the files.

S6, the terminal sends a file access request.

S7, the server receives a file access request sent by the terminal, and a target data center is determined in a preset data center according to the file access request.

And S7, the server acquires the stored file according to the determined target data center and sends the file to the terminal.

And S8, after the terminal sends the file access request, receiving the stored file sent by the server.

It can be understood that in the embodiment of the present invention, when a current file is stored, for a dual HBase cluster of a primary/standby/dual-active scheme, one copy of data is typically stored with 6 copies, which is a great waste for storage resources. In the invention, by deploying an HBase cluster, modifying a copy placement strategy of hdfs and introducing a containerization mode to deploy hmaster and regionserver of the HBase cluster, resources are saved, system complexity is simplified, and possibility of faults is reduced.

An embodiment of the present invention provides a data storage device, as shown in fig. 12, and fig. 12 is a schematic structural diagram of the data storage device provided in the embodiment of the present invention, where the data storage device is applied to a server, and the data storage device includes: a first receiving unit 1201, a first determining unit 1202, a first transmitting unit 1203; wherein,,

the first receiving unit 1201 is configured to receive a file upload request sent by a terminal.

The first sending unit 1202 is configured to determine, according to the file upload request, a storage location of the file by using a custom method, and send the file to a terminal.

The first receiving unit 1201 is further configured to receive a file access request sent by the terminal.

The first determining unit 1202 is configured to determine a target data center in a preset data center according to the file access request; the preset data center comprises a storage position of the file.

The first sending unit 1203 is further configured to obtain, through the target data center, the stored file according to the storage location, and send the file to the terminal.

In some embodiments of the invention, the preset data center comprises a first data center and a second data center; the first data center and the second data center each include a plurality of different racks; the plurality of different racks respectively comprise a plurality of different data nodes; taking the distance value between the data nodes of different racks of the first data center as a first distance value; the distance value between the data nodes of different racks of the second data center is a second distance value; the first determining unit 1202 is further configured to obtain a data node of the first data center and a data node of the second data center; and according to the first distance value and the second distance value, combining the data nodes of the first data center and the data nodes of the second data center, determining the storage position of the file and sending the storage position to the terminal.

In some embodiments of the present invention, the first determining unit 1202 is further configured to obtain a first data node; according to the first data node, combining the first distance value and the second distance value to respectively determine a second data node, a third data node and a fourth data node; and determining the storage position of the file according to the first data node, the second data node, the third data node and the fourth data node, and sending the file to the terminal.

In some embodiments of the present invention, the first determining unit 1202 is further configured to randomly acquire the first data node from the plurality of data nodes if the terminal is not in a cluster; the cluster comprises the plurality of data nodes; and if the terminal is in the cluster, the data node where the terminal is located is the first data node.

In some embodiments of the present invention, a sum of distances between any two data nodes among the first data node, the second data node, the third data node, and the fourth data node is a preset value, and the first determining unit 1202 is further configured to determine, if the first data node is a node of the first data center, the second data node according to the first data node and the first distance value; and determining the third data node according to the second data node and the second data center; the third data node is any node of the second data center; and determining a fourth data node according to the third data node and the second distance value;

In some embodiments of the present invention, a sum of distances between any two data nodes among the first data node, the second data node, the third data node, and the fourth data node is a preset value, and the first determining unit 1202 is further configured to determine, if the first data node is a node of the second data center, the second data node according to the first data node and the second distance value; and determining the third data node according to the second data node and the first data center; the third data node is any node of the first data center; and determining the fourth data node according to the third data node and the first distance value.

In some embodiments of the present invention, the first determining unit 1202 is further configured to compare distances between the first data center and the second data center and the terminal, respectively; determining a comparison result; determining the nearest data center according to the comparison result; if the latest data center is detected to be not faulty, acquiring the file according to the storage position of the file in the latest data center and sending the file to the terminal; and if the data center closest to the data center is detected to be faulty, acquiring the file according to the storage position of the file in another data center and sending the file to the terminal.

In some embodiments of the present invention, the first sending unit 1203 is further configured to compare the distances between the first data center and the second data center and the terminal, respectively; determining a comparison result; determining the data center closest to the data center according to the comparison result, wherein the data center closest to the data center is a target data center;

in some embodiments of the present invention, the first determining unit 1202 is further configured to, if the first data center is the target data center and it is detected that the first data center is not faulty, acquire the file according to the storage location in the first data center and send the file to the terminal; and if the first data center is the target data center and the first data center is detected to be faulty, acquiring the file according to the storage position in the second data center and sending the file to the terminal.

In some embodiments of the present invention, the first determining unit 1202 is further configured to restart the related service in the second data center when detecting that the first data center fails through the service layer identified by the deployed monitoring; and in a second data center after the related service is started, acquiring the file according to the storage position and sending the file to the terminal.

It can be understood that in the implementation scheme of the device, the server receives the file uploading request sent by the terminal, determines the storage position of the file by using a self-defining method and sends the file to the terminal; after receiving the file access request sent by the terminal, determining a target data center in a preset data center according to the file access request, and finally obtaining a stored file through the target data center and sending the stored file to the terminal. The related service of the cluster is deployed by a user-defined copy placement strategy for reforming hdfs and a containerization mode, so that the resources are saved, the complexity of the system is simplified, and the possibility of faults is reduced.

An embodiment of the present invention provides a data storage device, as shown in fig. 13, and fig. 13 is a schematic structural diagram of another data storage device provided in an embodiment of the present invention, where the data storage device is applied to a terminal, and the data storage device includes: a second transmitting unit 1301 and a second receiving unit 1302; wherein,,

the second sending unit 1301 is configured to send a file upload request to a server.

The second receiving unit 1302 is configured to receive, according to the file upload request, a storage location of the file sent by the server.

The second sending unit 1301 is further configured to send, according to the received storage locations, the files to the storage locations in order for storage.

The second sending unit 1301 is further configured to send a file access request to the server if the file stored by the terminal is damaged.

The second receiving unit 1302 is further configured to receive, based on the file access request, the file sent by the server through the storage location for replacement.

It can be understood that in the implementation scheme of the device, the terminal performs file storage according to the number of the stored files sent by the server and different from the number of the existing stored files by sending a file uploading request to the server, and utilizes the sent file access request, so that the fault tolerance of file storage is improved, and storage and resources are saved.

Based on the method of the foregoing embodiment, as shown in fig. 14, a schematic structural diagram of a server provided in the embodiment of the present invention, fig. 14 is a schematic structural diagram of a data storage device provided in the embodiment of the present application, where the server 14 includes: a first processor 1401 and a first memory 1402; the first memory 1402 stores one or more programs executable by the first processor 1401, and when the one or more programs are executed, a method of data storage corresponding to the server of the foregoing embodiment is performed by the first processor 1401.

Based on the method of the foregoing embodiment, as shown in fig. 15, a schematic structural diagram of a terminal provided in the embodiment of the present invention, fig. 15 is a schematic structural diagram of a data storage device provided in the embodiment of the present application, where the terminal 15 includes: a second processor 1501 and a second memory 1502; the second memory 1502 stores one or more programs executable by the second processor 1501, and when the one or more programs are executed, a method of data storage corresponding to the terminal of the foregoing embodiment is performed by the second processor 1501.

The embodiment of the invention provides a computer readable storage medium, which is applied to a server and is used for storing executable instructions for realizing a method described by the server when a first processor is caused to execute.

The embodiment of the invention provides a computer readable storage medium which is applied to a terminal and is stored with executable instructions for realizing the method corresponding to the terminal when a second processor is caused to execute.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention.

Claims

1. A data storage method, applied to a server, comprising:

receiving a file uploading request sent by a terminal;

determining a storage position of a file by using a self-defining method according to the file uploading request and sending the storage position to the terminal;

receiving a file access request sent by the terminal;

2. The method of claim 1, wherein the preset data center comprises a first data center and a second data center; the first data center and the second data center each include a plurality of different racks; the plurality of different racks respectively comprise a plurality of different data nodes; taking the distance value between the data nodes of different racks of the first data center as a first distance value; the distance value between the data nodes of different racks of the second data center is a second distance value;

and according to the first distance value and the second distance value, combining the data nodes of the first data center and the data nodes of the second data center, determining the storage position of the file and transmitting the file to a terminal.

3. The method according to claim 2, wherein the determining the storage location of the file according to the first distance value and the second distance value in combination with the data node of the first data center and the data node of the second data center and sending the determined storage location to the terminal includes:

Acquiring a first data node;

4. A method according to claim 3, wherein said obtaining a first data node comprises:

traversing a plurality of data nodes in a cluster, and randomly acquiring the first data node from the plurality of data nodes if the terminal is not in the cluster;

5. A method according to claim 3, wherein the sum of the distances between any two of the first data node, the second data node, the third data node and the fourth data node is a first distance value;

according to the first data node, combining the first distance value and the second distance value, determining a second data node, a third data node and a fourth data node respectively, including:

and determining the fourth data node according to the third data node and the second distance value.

6. A method according to claim 3, wherein said determining, from said first data node, in combination with said first distance value and said second distance value, a second data node, a third data node and a fourth data node, respectively, comprises:

7. The method of claim 1, wherein determining a target data center among preset data centers based on the file access request comprises:

determining the nearest data center according to the comparison result;

and taking the nearest data center as the target data center.

8. The method of claim 7, wherein the obtaining, by the target data center, the stored file according to the storage location and sending the file to the terminal comprises:

9. The method according to claim 8, wherein if the first data center is the target data center and the first data center is detected to be faulty, acquiring the file according to the storage location in the second data center and sending the file to the terminal includes:

10. A data storage method, applied to a terminal, comprising:

sending a file uploading request to a server;

receiving a storage position of a file sent by the server according to the file uploading request;

and according to the received storage positions, sequentially sending the files to the storage positions for storage.

11. The method of claim 10, wherein after sequentially sending the files to the storage location for storage according to the storage location, the method further comprises:

if the file stored by the terminal is damaged; sending a file access request to the server;

receiving the file stored in the storage position and sent by the server according to the file access request;

and replacing the file stored by the terminal by using the file.

12. A data storage device, comprising:

13. A data storage device, comprising:

the second receiving unit is used for receiving the storage position of the file sent by the server according to the file uploading request;

14. A server, the server comprising:

a first memory for storing executable instructions;

a first processor for implementing the method of any of claims 1-9 when executing executable instructions stored in the first memory.

15. A terminal, the terminal comprising:

a second memory for storing executable instructions;

a second processor for implementing the method of any one of claims 10 or 11 when executing executable instructions stored in said second memory.

16. A computer readable storage medium storing executable instructions which, when executed, are adapted to cause a first processor to perform the data storage method of any one of claims 1-9; or for causing a second processor to perform the data storage method of any one of claims 10 or 11.