WO2016101759A1

WO2016101759A1 - Data routing method, data management device and distributed storage system

Info

Publication number: WO2016101759A1
Application number: PCT/CN2015/095507
Authority: WO
Inventors: 陈营; 李明昊; 宋昭; 陈宗志; 王超
Original assignee: 北京奇虎科技有限公司; 奇智软件（北京）有限公司
Priority date: 2014-12-27
Filing date: 2015-11-25
Publication date: 2016-06-30
Also published as: CN104580428A; CN104580428B

Abstract

Provided are a data routing method, a data management device and a distributed storage system. The method specifically comprises: a data node receiving a metadata routing request from a client, wherein the metadata routing request carries first keyword information about data; the data node obtaining data node information corresponding to the first keyword information according to a metadata table matching the first keyword information itself, wherein the data node information corresponding to the keyword is stored in the metadata table, and the metadata table is obtained based on the maintenance of the communication between data nodes; and the data node returning the data node information to the client. The embodiments of the present invention can guarantee that correct data node information is fed back to the client, thereby increasing the accuracy rate of metadata routing, and furthermore, the operation and maintenance costs of a distributed storage system can be reduced.

Description

Data routing method, data management device and distributed storage system

Technical field

The present invention relates to the field of distributed storage technologies, and in particular, to a data routing method, a data management apparatus, and a distributed storage system.

Background technique

GFS (Google File System) is a large distributed file system that provides massive storage for Google Cloud Computing and is tightly integrated with MapReduce (Map Simplification) technology.

Referring to FIG. 1, a schematic structural diagram of a GFS in the prior art is shown. GFS divides nodes of the entire system into three types of roles: Client (Client), Master (Master Server), and Chunk Server (Data Block Server); Client is the access interface provided by GFS to the application; Master is the management node of GFS, there is only one logic, it saves the metadata of the system and is responsible for the management of the entire file system; Chunk Server is responsible for the specific storage work, the data is filed. The form is stored on the Chunk Server, and the number of Chunk Servers can be multiple.

When the client accesses the GFS, the client first sends a metadata routing request to the master, where the metadata routing request carries key (keyword) information, and the master obtains the Chunk Server information corresponding to the key according to the stored metadata table, and then directly Access these Chunk Servers to complete data access. This design method of GFS realizes the separation of control flow and data flow. There is only control flow between the Client and the Master, and there is no data flow, which greatly reduces the load of the Master, so that it does not become a bottleneck of system performance. The client and the Chunk Server directly transmit data streams. At the same time, because the files are divided into multiple Chunk for distributed storage, the Client can access multiple Chunk Servers at the same time, so that the I/O of the entire system is highly parallel and the overall performance of the system is improved.

However, factors such as network jitter and node failure may cause the Chunk server corresponding to the key to change. The master as the management node cannot know the change of the metadata in the above situation in time. Therefore, the correct Chunk Server information cannot be guaranteed to the client. Normally, when the metadata request fails, the client also needs to send a request to update the metadata table to the master, and continues to wait for the Chunk Server information returned by the master according to the updated metadata table.

Summary of the invention

In view of the above problems, the present invention has been made in order to provide a data routing method, a data management apparatus, and a distributed storage system that overcome the above problems or at least partially solve the above problems.

According to an aspect of the present invention, a data routing method is provided, including:

Receiving, by the data node, a metadata routing request from the client, where the metadata routing request carries first keyword information of the data;

The data node obtains data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information; wherein the metadata table stores data node information corresponding to the keyword, The metadata table is maintained based on communication between data nodes;

The data node returns the data node information to the client.

According to another aspect of the present invention, a computer program is provided comprising computer readable code that, when executed on a computing device, causes the computing device to perform a data routing method as described above.

According to still another aspect of the present invention, there is provided a computer readable medium storing a computer program as described above.

According to still another aspect of the present invention, a data management apparatus is provided, including:

a first receiving module, configured to receive a metadata routing request from the client, where the metadata routing request carries first keyword information of the data;

The first matching module is configured to obtain data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information, where the metadata table stores data corresponding to the keyword Node information, which is obtained based on communication maintenance between data nodes;

a first return module configured to return the data node information to the client.

According to an embodiment of the present invention, a data routing method, a data management apparatus, and a distributed storage system, a data node can process metadata of a client by using a metadata table stored by itself. Routing request; since the data node can maintain the metadata table based on the communication between the data nodes, the maintained metadata table can reflect the change of the node state in time, thereby ensuring that the correct data node information is fed back to the client, thereby improving The accuracy of the metadata routing; and the use of the dedicated master to store and maintain the metadata table acquisition in the prior art, the embodiment of the present invention reduces the role of the master, thereby reducing the operation and maintenance deployment of the distributed storage system. cost.

The above description is only an overview of the technical solutions of the present invention, and the above-described and other objects, features and advantages of the present invention can be more clearly understood. Specific embodiments of the invention are set forth below.

DRAWINGS

Various other advantages and benefits will become apparent to those skilled in the art from a The drawings are only for the purpose of illustrating alternative embodiments and are not to be considered as limiting. Throughout the drawings, the same reference numerals are used to refer to the same parts. In the drawing:

1 is a schematic structural view of a prior art GFS;

2 is a flow chart showing the steps of a data routing method according to an embodiment of the present invention;

3 is a block diagram showing the structure of a distributed storage system according to an example of the present invention;

FIG. 4 is a flow chart showing the steps of a data routing method according to an embodiment of the present invention;

FIG. 5 is a flow chart showing the steps of a data routing method according to an embodiment of the present invention;

6 is a flow chart showing the steps of maintaining a data table based on communication between data nodes according to an embodiment of the present invention;

FIG. 7 is a flow chart showing the steps of a data routing method according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a data management apparatus according to an embodiment of the present invention; FIG.

FIG. 9 is a schematic structural diagram of a data management apparatus according to an embodiment of the present invention; FIG.

FIG. 10 is a schematic structural diagram of a data management apparatus according to an embodiment of the present invention; FIG.

FIG. 11 is a schematic structural diagram of a distributed storage system according to an embodiment of the present invention; FIG.

Figure 12 schematically shows a block diagram of a computing device for performing the method according to the invention;

Fig. 13 schematically shows a storage unit for holding or carrying program code implementing the method according to the invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While the embodiments of the present invention have been shown in the drawings, the embodiments Rather, these embodiments are provided so that this disclosure will be more fully understood and the scope of the disclosure will be fully disclosed.

Referring to FIG. 2, a schematic flowchart of a step of a data routing method according to an embodiment of the present invention is shown, which may include the following steps:

Step 201: The data node receives a metadata routing request from the client, where the metadata routing request carries the first keyword information of the data.

Step 202: The data node matches the metadata table of the first keyword information to obtain data node information corresponding to the first keyword information, where the metadata table stores data corresponding to the keyword. Node information, which is obtained based on communication maintenance between data nodes;

Step 203: The data node returns the data node information to the client.

The embodiments of the present invention can be applied to distributed storage systems of various services for improving the accuracy of metadata routing.

Referring to FIG. 3, a schematic structural diagram of a distributed storage system according to an example of the present invention is shown. Specifically, the method includes: a client 301 and a data node 302. The client 301 serves as an initiator of a service request and can pass the element. The data routing request acquires the data node information corresponding to the metadata routing request from the data node 302, so that the data node information corresponding data node 302 can be accessed to complete the data access operation; the data node 302 can store the metadata. Table, And the data nodes 302 can communicate with each other, so that the metadata table can be maintained based on communication between the data nodes 302 to ensure that the metadata table can reflect changes in node status in time.

In an optional embodiment of the present invention, the step of the data node obtaining the data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information may specifically include :

Sub-step A1, calculating a hash value of the first keyword information;

Sub-step A2, matching the metadata table according to the hash value, to obtain the hash value corresponding data node information.

In a specific implementation, a hash algorithm such as a one-way hash algorithm may be used to calculate a hash value of the first keyword information, and the specific hash algorithm is not limited in the embodiment of the present invention.

In an actual application, the data node information may specifically include one or more of the following information: node number information, node attribute information, and node communication rate information. The node attribute information may specifically include: information that the node is available or the node is unavailable, and the node communication rate information may be a communication rate value of the node or the like.

In an application example of the present invention, the node number information of the data node corresponding to the hash value may be the number of the data node itself, or may be the number of other data nodes, whichever is the case, the data node The hash value corresponding to the node number of the data node may be returned to the client to satisfy the client's further node access request.

In summary, the data node can use its stored metadata table to process the client's metadata routing request; since the data node can maintain the metadata table based on the communication between the data nodes, the maintained metadata table can reflect the node in time. The change of the state, therefore, can ensure that the correct data node information is fed back to the client, and the accuracy of the metadata routing is improved; and the implementation of the present invention is implemented by using a dedicated Master to store and maintain the metadata table acquisition in the prior art. This reduces the role of the Master, thus reducing the operating and deployment costs of distributed storage systems.

Referring to FIG. 4, a schematic flowchart of a step of a data routing method according to an embodiment of the present invention is shown, which may include the following steps:

Step 401: The data node receives a metadata routing request from a client, where the element The data routing request carries the first keyword information of the data;

Step 402: The data node matches the metadata table of the first keyword information to obtain data node information corresponding to the first keyword information, where the metadata table stores data corresponding to the keyword. Node information, which is obtained based on communication maintenance between data nodes;

Step 403: The data node returns the data node information to the client.

Step 404: When the data node information corresponding to the first keyword information is not the data node itself, the data node returns the metadata table to the client.

With respect to the embodiment shown in FIG. 2, in the embodiment, when the data node information corresponding to the first keyword information is not the data node itself, the data node may further return the metadata table to the client. End, that is, the current metadata table can be provided to the client, so that the client can query the metadata by querying its own metadata table.

For the client, the metadata can be routed in the following two ways: the first method is to perform metadata routing by sending a metadata routing request to the data node, and the second method is to query the metadata table of the user by querying its own metadata table. The routing of metadata, in which the first method has the advantage of high accuracy, the second method has the advantage of saving traffic. It can be understood that any one or both of the above two methods may be determined by a person skilled in the art according to actual needs. For example, when the accuracy requirement is strict, the foregoing method 1 may be adopted, and, for example, the traffic demand is strict. In the above manner, the second method may be adopted, or in order to ensure the success rate of the route, the foregoing manners 1 and 2 may be simultaneously adopted.

Referring to FIG. 5, a schematic flowchart of a step of a data routing method according to an embodiment of the present invention is shown, which may specifically include the following steps:

Step 501: The data node receives a metadata routing request from the client, where the metadata routing request carries the first keyword information of the data.

Step 502: The data node matches the metadata table of the first keyword information to obtain data node information corresponding to the first keyword information. The metadata table stores data corresponding to the keyword. Node information, which is based on communication maintenance between data nodes get;

Step 503: The data node returns the data node information to the client.

Step 504: The data node performs maintenance on the metadata table based on communication between data nodes.

Referring to FIG. 6 , a flow chart of steps for maintaining a data table based on communication between data nodes is performed according to an embodiment of the present invention, which may specifically include the following steps:

Sub-step 541, the coordinator node sends a preparation message to all participant nodes executing the transaction T;

Sub-step 542, each participant node determines whether to submit a transaction T, and if so, returns a ready message to the coordinator node; otherwise, returns an abnormal abort message to the coordinator node;

Sub-step 543, when the messages obtained from all participant nodes are ready messages, the coordinator node sends a formal commit message to all participant nodes;

Sub-step 544, after receiving the formal submission message, each participant node officially completes the transaction T, releases the occupation during the entire transaction T, and sends a completion message to the coordinator node;

Sub-step 545, the coordinator node completes the transaction after receiving the completion message fed back by all the participant nodes;

Sub-step 546, when the coordinator node has an abnormal abort message in the message obtained from all the participant nodes, or when the response message of all the participant nodes cannot be acquired before the timeout, the coordinator node to all the participant nodes Issue a rollback message;

Here, the timeout can indicate that the official submission message has been issued until the preset period is exceeded.

Sub-step 547, after receiving the rollback message, each participant node performs a rollback operation on the transaction T, releases the resources occupied during the entire T transaction, and sends a rollback completion message to the coordinator node;

Sub-step 548, after the coordinator node receives the rollback completion message fed back by all the participant nodes, cancels the transaction T.

With respect to the embodiment shown in FIG. 1, the data node of this embodiment may specifically include: a coordinator node and a participant node, and adding that the data node maintains the metadata table based on communication between data nodes. Wherein, the metadata maintenance involved in the above sub-step 541 - sub-step 548 The process specifically guarantees the consistency and integrity of the metadata tables maintained by all participant nodes through a two-phase commit protocol.

The main idea of the two-phase commit protocol can be: when a transaction T is to operate on multiple databases, it must be ensured that the database of the multiple participant nodes is successfully submitted, and the transaction T can succeed. Therefore, the coordination node can firstly The participant node issues a pre-submission, and the multiple participant nodes return whether they can submit. If all the participant nodes can submit, the coordination node can formally submit the transaction T.

In the metadata maintenance process involved in the above sub-step 541 - sub-step 548, sub-step 541 - sub-step 542 is the first stage step, and sub-step 543 - sub-step 548 is the second stage step. Wherein, in the first phase, the coordinator node will notify the participant node of the transaction to prepare to commit or cancel the transaction, and the participant node informs the coordinator node of its decision: consent (ready message) or cancellation (abnormal abort message); In the second phase, the coordinator node will make decisions based on information fed back by all participant nodes: commit or cancel, and the coordinator node notifies all participant nodes to commit the transaction if and only if all participant nodes agree to commit the transaction. Otherwise, the coordinator node will notify all participant nodes to cancel the transaction.

It should be noted that the above scheme for ensuring the consistency and integrity of the metadata table maintained by all participant nodes through the two-phase commit protocol is only as a guarantee for the consistency and integrity of the metadata table maintained by all participant nodes. Alternatively, it can be understood that those skilled in the art can adopt other schemes to ensure the consistency and integrity of the metadata table maintained by all participant nodes according to actual needs, such as a three-phase commit protocol, etc. The specific scheme for ensuring the consistency and integrity of the metadata tables maintained by all participant nodes is not limited.

Referring to FIG. 7, a flow chart of steps of a data routing method according to an embodiment of the present invention is shown, which may specifically include the following steps:

Step 701: The data node receives a metadata routing request from the client, where the metadata routing request carries the first keyword information of the data.

Step 702: The data node matches the metadata table of the first keyword information to obtain data node information corresponding to the first keyword information, where the metadata table stores Data node information corresponding to the keyword, the metadata table is obtained based on communication maintenance between the data nodes;

Step 703: The data node returns the data node information to the client.

Step 704: The data node receives a read request from a client, where the read request carries second keyword information of data.

Step 705: The data node matches its own metadata table according to the second keyword information carried by the read request, to obtain data node information corresponding to the second keyword information.

Step 706: The data node determines, according to the data node information corresponding to the second keyword information, whether the read request corresponding data is in the data node itself;

Step 707: When the data node information corresponding to the second keyword information is the data node itself, the data node queries its own data engine according to the read request, and returns the data obtained by the query to the client. end;

Step 708: When the data node information corresponding to the second keyword information is not the data node itself, forwarding the read request to the first data node corresponding to the data node information corresponding to the second keyword information;

Step 709: Receive data corresponding to the read request returned by the first data node, and return the data to the client.

With respect to the embodiment shown in FIG. 1, in addition to processing the metadata routing request from the client by performing steps 701-step 703, the embodiment may also process the read request from the client through steps 704-step 709, in particular, In the process of processing the read request, when the data node information corresponding to the second keyword information carried in the read request is not the data node itself, the data node may also play the role of a network proxy, that is, And forwarding the read request to the first data node corresponding to the data node information corresponding to the second keyword information, and the data corresponding to the read request returned by the first data node is returned to the client; The process in which the proxy client forwards the read request and returns the read data to the client can prevent the client from attempting to send a read request to multiple different data nodes, thereby saving the traffic of the client.

It should be noted that, in addition to the proxy client forwarding the read request and returning the read data to the client, the data node may also forward the write request to the client and return the response result to the client. The process of forwarding the write request to the proxy client is similar to the process of forwarding the read request by the proxy client, so it will not be described here and cross-referenced.

For the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the embodiments of the present invention are not limited by the described action sequence, because the embodiment according to the present invention Some steps can be performed in other orders or at the same time. In the following, those skilled in the art should also understand that the embodiments described in the specification are optional embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

FIG. 8 is a schematic structural diagram of a data management apparatus according to an embodiment of the present invention, which may specifically include the following modules:

The first receiving module 801 is configured to receive a metadata routing request from the client, where the metadata routing request carries the first keyword information of the data;

The first matching module 802 is configured to obtain data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information, where the metadata table stores a keyword corresponding Data node information obtained based on communication maintenance between data nodes;

The first returning module 803 is configured to return the data node information to the client.

In an actual application, the data management device may be a data node itself or a device for managing each data node. The specific location of the data management device is not limited in the embodiment of the present invention.

In an optional embodiment of the present invention, the first matching module 802 may specifically include:

a hash calculation submodule configured to calculate a hash value of the first keyword information; and

The hash matching submodule is configured to match the metadata table according to the hash value to obtain the hash value corresponding data node information.

In another optional embodiment of the present invention, the data node may further include:

And a second returning module, configured to: when the data node information corresponding to the first keyword information is not the data node itself, the data node returns the metadata table to the client.

9 is a schematic structural diagram of a data management apparatus according to an embodiment of the present invention, which may specifically include: a coordinator node 901 and a participant node 902;

The coordinator node 901 may specifically include: a first maintenance module 911 configured to perform maintenance on the metadata table based on communication between data nodes;

The participant node 902 may specifically include the following modules:

The first receiving module 921 is configured to receive a metadata routing request from the client, where the metadata routing request carries the first keyword information of the data;

The first matching module 922 is configured to obtain the data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information, where the metadata table stores the keyword corresponding Data node information, which is obtained based on communication maintenance between data nodes;

a first returning module 923, configured to return the data node information to the client; and

The second maintenance module 924 is configured to perform maintenance on the metadata table based on communication between data nodes;

The first maintenance module 911 may specifically include:

Preparing to send a sub-module 9111 configured to send a preparation message to all participant nodes executing the transaction T;

The formal submission sending sub-module 9112 is configured to issue a formal submission message to all participant nodes when the messages obtained from all the participant nodes are ready messages;

The transaction completion sub-module 9113 is configured to complete the transaction after receiving the completion message fed back by all the participant nodes;

Rollback sending sub-module 9114, configured to have an abort message in a message obtained from all participant nodes, or to obtain a response message of all participant nodes before timing out, the coordinator node to all participant nodes Issue a rollback message; and

The transaction cancellation sub-module 9115 is configured to cancel the transaction T after receiving the rollback completion message fed back by all the participant nodes;

The second maintenance module 924 may specifically include:

Preparing a response sub-module 9241, configured to determine whether to submit a transaction T, and if so, to the association The caller node returns a ready message, otherwise, returns an abort message to the coordinator node;

The transaction execution sub-module 9242 is configured to formally complete the transaction T after receiving the formal submission message, release the occupation during the entire transaction T, and send the completion message to the coordinator node;

The rollback submodule 9243 is configured to perform, by each participant node, a rollback operation on the transaction T after receiving the rollback message, release the resources occupied during the entire T transaction, and send back to the coordinator node. Roll the completion message.

Referring to FIG. 10, a schematic structural diagram of a data management apparatus according to an embodiment of the present invention is shown, which may specifically include the following modules:

The first receiving module 1001 is configured to receive a metadata routing request from the client, where the metadata routing request carries first keyword information of the data;

The first matching module 1002 is configured to obtain data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information, where the metadata table stores a keyword corresponding Data node information, which is obtained based on communication maintenance between data nodes;

The first returning module 1003 is configured to return the data node information to the client;

The second receiving module 1004 is configured to receive a read request from the client, where the read request carries the second keyword information of the data;

The second matching module 1005 is configured to match the metadata table of the second keyword information carried by the read request to obtain the data node information corresponding to the second keyword information;

The determining module 1006 is configured to determine, according to the data node information corresponding to the second keyword information, whether the read request corresponding data is in the data node itself;

The query module 1007 is configured to: when the data node information corresponding to the second keyword information is the data node itself, the data node queries its own data engine according to the read request, and returns the data obtained by the query to The client;

The forwarding module 1008 is configured to: when the data node information corresponding to the second keyword information is not the data node itself, forward the read request to the first data corresponding to the data node information corresponding to the second keyword information Node;

The third returning module 1009 is configured to receive data corresponding to the read request returned by the first data node, and return the data to the client.

The present invention also provides a distributed storage system, which may specifically include: a client and the foregoing data management device.

FIG. 11 is a schematic structural diagram of a distributed storage system according to an embodiment of the present invention, which may specifically include: a client 1101 and a plurality of data nodes 1102;

The data node 1102 may specifically include the following modules:

The first receiving module 1121 is configured to receive a metadata routing request from the client, where the metadata routing request carries the first keyword information of the data;

The first matching module 1122 is configured to obtain data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information, where the metadata table stores a keyword corresponding Data node information obtained based on communication maintenance between data nodes;

The first return module 1123 is configured to return the data node information to the client.

For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The various component embodiments of the present invention may be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the components of the data routing method, data management apparatus, and distributed storage system in accordance with embodiments of the present invention. Some or all of the features. The invention can also be implemented as a device or device program (e.g., a computer program and a computer program product) for performing some or all of the methods described herein. Such a program implementing the invention may be stored on a computer readable medium or may be in the form of one or more signals. Such signals may be downloaded from an internet platform, provided on a carrier signal, or provided in any other form.

For example, Figure 12 illustrates a computing device, such as a search, that can implement the above method in accordance with the present invention. Cable engine server. The computing device conventionally includes a processor 1210 and a computer program product or computer readable medium in the form of a memory 1230. The memory 1230 may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), an EPROM, a hard disk, or a ROM. Memory 1230 has a storage space 1250 that stores program code 1251 for performing any of the method steps described above. For example, storage space 1250 storing program code may include various program codes 1251 for implementing various steps in the above methods, respectively. The program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. Such a computer program product is typically a portable or fixed storage unit such as that shown in FIG. The storage unit may have storage segments, storage spaces, and the like that are similarly arranged to memory 1230 in the computing device of FIG. The program code can be compressed, for example, in an appropriate form. Typically, the storage unit comprises computer readable code 1251' for performing the steps of the method according to the invention, ie code that can be read by a processor such as 1210, which when executed by the server causes the server to execute Each step in the described method.

"an embodiment," or "an embodiment," or "an embodiment," In addition, it is noted that the phrase "in one embodiment" is not necessarily referring to the same embodiment.

In the description provided herein, numerous specific details are set forth. However, it is understood that the embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of the description.

It is to be noted that the above-described embodiments are illustrative of the invention and are not intended to be limiting, and that the invention may be devised without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as a limitation. The word "comprising" does not exclude the presence of the elements or steps that are not recited in the claims. The word "a" or "an" The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means can be embodied by the same hardware item. The use of the words first, second, and third does not indicate any order. These words can be interpreted as names.

In addition, it should be noted that the language used in the specification has been selected for the purpose of readability and teaching, and is not intended to be construed or limited. Therefore, many modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. The disclosure of the present invention is intended to be illustrative, and not restrictive, and the scope of the invention is defined by the appended claims.

Claims

A data routing method includes:

Receiving, by the data node, a metadata routing request from the client, where the metadata routing request carries first keyword information of the data;

The data node obtains data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information; wherein the metadata table stores data node information corresponding to the keyword, The metadata table is maintained based on communication between data nodes;

The data node returns the data node information to the client.
The method of claim 1 wherein the method further comprises:

When the data node information corresponding to the first keyword information is not the data node itself, the data node returns the metadata table to the client.
The method of claim 1 or 2, wherein the data node comprises: a coordinator node and a participant node;

The method further includes: the data node maintaining the metadata table based on communication between the data nodes;

The step of maintaining, by the data node, the metadata table based on communication between data nodes, including:

The coordinator node sends a preparation message to all participant nodes that execute the transaction T;

Each participant node determines whether to submit a transaction T, and if so, returns a ready message to the coordinator node, otherwise returns an abnormal abort message to the coordinator node;

When the messages obtained from all participant nodes are ready messages, the coordinator node issues a formal commit message to all participant nodes;

After receiving the formal submission message, each participant node officially completes the transaction T, releases the occupation during the entire transaction T period, and sends a completion message to the coordinator node;

The coordinator node completes the transaction after receiving the completion message fed back by all the participant nodes;

When there is an abnormal abort message in the message obtained from all the participant nodes, or when the response message of all the participant nodes cannot be acquired before the timeout, the coordinator node issues a rollback message to all the participant nodes;

After receiving the rollback message, each participant node performs a rollback operation on the transaction T, releases the resources occupied during the entire T transaction, and sends a rollback completion message to the coordinator node;

The coordinator node cancels the transaction T after receiving the rollback completion message fed back by all the participant nodes.
The method of claim 1 or 2 or 3, wherein the method further comprises:

Receiving, by the data node, a read request from a client, where the read request carries second keyword information of data;

The data node matches the metadata table of the second key information carried by the read request to obtain the data node information corresponding to the second keyword information;

Determining, according to the data node information corresponding to the second keyword information, whether the read request corresponding data is in the data node itself;

When the data node information corresponding to the second keyword information is the data node itself, the data node queries its own data engine according to the read request, and returns the data obtained by the query to the client;

When the data node information corresponding to the second keyword information is not the data node itself, forwarding the read request to the first data node corresponding to the data node information corresponding to the second keyword information;

Receiving data corresponding to the read request returned by the first data node, and returning the data to the client.
The method of claim 1 or 2 or 3, wherein the data node obtains data node information corresponding to the first keyword information according to the first keyword information matching its own metadata table, include:

Calculating a hash value of the first keyword information;

Matching the metadata table according to the hash value, and obtaining data node information of the data node corresponding to the hash value.
The method of claim 1 or 2 or 3, wherein the data node information comprises one or more of the following information: node number information, node attribute information, and node communication rate information.
A computer program comprising computer readable code when said computer readable code is When the computing device is running, the computing device is caused to perform the data routing method according to any one of claims 1 to 6.
A computer readable medium storing the computer program of claim 7.
A data management device comprising:

a first receiving module, configured to receive a metadata routing request from the client, where the metadata routing request carries first keyword information of the data;

The first matching module is configured to obtain data node information corresponding to the first keyword information according to the first keyword information matching the metadata table of the first keyword information, where the metadata table stores data corresponding to the keyword Node information, which is obtained based on communication maintenance between data nodes;

a first return module configured to return the data node information to the client.
The data management device of claim 9, wherein the data management device further comprises:

And a second returning module, configured to: when the data node information corresponding to the first keyword information is not the data node itself, the data node returns the metadata table to the client.
The data management device according to claim 9 or 10, wherein said data management device comprises: a coordinator node and a participant node;

The coordinator node includes: a first maintenance module configured to maintain the metadata table based on communication between the data nodes;

The participant node includes: the foregoing first receiving module, a first matching module, a first returning module, and a second maintenance module configured to perform maintenance on the metadata table based on communication between data nodes;

The first maintenance module includes:

Preparing to send a submodule configured to send a preparation message to all participant nodes that execute transaction T;

Formally submit a sending sub-module configured to issue a formal commit message to all participant nodes when the messages obtained from all participant nodes are ready messages;

The transaction completion sub-module is configured to complete the transaction after receiving the completion message fed back by all the participant nodes;

Rollback the sending submodule configured to have an abort message in the message obtained from all the participant nodes, or to send a response message of all the participant nodes before the timeout, the coordinator node issues to all the participant nodes Rollback message; and

The transaction canceling submodule is configured to cancel the transaction T after receiving the rollback completion message fed back by all the participant nodes;

The second maintenance module includes:

Preparing a response sub-module configured to determine whether to commit the transaction T, and if so, returning a ready message to the coordinator node; otherwise, returning an abort message to the coordinator node;

a transaction execution submodule configured to formally complete the transaction T after receiving the formal commit message, release the occupation during the entire transaction T, and send a completion message to the coordinator node;

Rollback sub-module, configured, after receiving the rollback message, each participant node performs a rollback operation on the transaction T, releases the resources occupied during the entire T transaction, and sends a rollback to the coordinator node. Complete the message.
The data management device according to claim 9 or 10 or 11, wherein the data management device further comprises:

a second receiving module, configured to receive a read request from the client, where the read request carries second keyword information of the data;

The second matching module is configured to match the metadata table of the second keyword information that is carried by the read request to obtain the data node information corresponding to the second keyword information;

The determining module is configured to determine, according to the data node information corresponding to the second keyword information, whether the read request corresponding data is in the data node itself;

a query module, configured to: when the data node information corresponding to the second keyword information is the data node itself, the data node queries its own data engine according to the read request, and returns the data obtained by the query to the Client

a forwarding module, configured to forward the read request to a data node information corresponding to the second keyword information when the data node information corresponding to the second keyword information is not the data node itself Corresponding first data node;

The third returning module is configured to receive data corresponding to the read request returned by the first data node, and return the data to the client.
The data management device of claim 9 or 10 or 11, wherein the first matching module comprises:

a hash calculation submodule configured to calculate a hash value of the first keyword information; and

The hash matching submodule is configured to match the metadata table according to the hash value to obtain data node information of the data node corresponding to the hash value.
The data management device according to claim 9 or 10 or 11, wherein said data node information comprises one or more of the following: node number information, node attribute information, and node communication rate information.
A distributed storage system comprising: a client and the data management device according to any one of claims 9 to 14.