CN105069152B - data processing method and device - Google Patents
data processing method and device Download PDFInfo
- Publication number
- CN105069152B CN105069152B CN201510524829.6A CN201510524829A CN105069152B CN 105069152 B CN105069152 B CN 105069152B CN 201510524829 A CN201510524829 A CN 201510524829A CN 105069152 B CN105069152 B CN 105069152B
- Authority
- CN
- China
- Prior art keywords
- management node
- management
- information
- metadata
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/1827—Management specifically adapted to NAS
Abstract
The embodiment of the invention provides a data processing method, which comprises the following steps: A. any management node receives a request for updating metadata; B. any management node judges whether the management node is a main management node, if so, the step C is carried out, if not, a request for updating the metadata is sent to the main management node, and the step C is carried out; C. and the master management node sends a metadata updating request to all the slave management nodes so as to update all the management nodes in the cross-data center storage architecture according to the metadata information to be updated. By adopting the method and the device provided by the embodiment of the invention, when the metadata information needs to be updated, the main management node and the slave management node update the stored metadata information, so that the metadata information stored by the main management node and the slave management node is the same, and when any management node fails, the metadata information stored by other management nodes is the same, so that the idle time in the prior art can not occur.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
background
with the development of computer technology, the data volume in various fields tends to increase in a massive manner, the demand for data storage is higher and higher, and the requirement for data backup is also improved. For example, data applied to satellite remote sensing and the like is developing from PB level to EB level, and how to meet the requirement of EB level data storage becomes an urgent problem to be solved for a large data storage platform.
currently, multiple data centers are the main method for solving the big data challenge, and meet the requirement of data capacity expansion, for example, companies such as satellite application, Google, Facebook and the like establish multiple data centers globally. For the management inconvenience of data by multiple data centers, a cross-data center management architecture, such as the spider management system of Google, is proposed. Management nodes across a data center management architecture sometimes fail, such as being powered down.
There are two countermeasures for this: firstly, an NFS (Network File System) remote mount directory stores a plurality of copies, local metadata information is synchronized to a remote File server, and when a fault occurs, the information on the remote File server can be synchronized to a normally operating management node; and secondly, the main management node sends the information stored by the main management node to the backup management node at regular time, and when the main management node fails, the backup management node replaces the main management node.
in the process of realizing the creation of the invention, the inventor finds that an idle time exists in the processes of synchronizing the information on the remote file server to the normally operating management node and replacing the main management node by the backup management node, and no management node stores the metadata information updated in the idle time.
disclosure of Invention
in view of the above, the present invention provides a data processing method and apparatus, so as to solve the problem in the prior art that no management node stores metadata information updated or modified in the period of time during the idle time, and a technical solution thereof is as follows:
A data processing method is applied to a cross-data center storage architecture, the cross-data center storage architecture comprises at least two management nodes, one management node of the at least two management nodes is used as a master management node, and the other management nodes are used as slave management nodes, and the data processing method comprises the following steps:
A. any management node receives a request for updating metadata, wherein the request for updating the metadata comprises metadata information to be updated;
B. The management node judges whether the management node is a main management node, if so, the step C is carried out, otherwise, the request for updating the metadata is sent to the main management node, and the step C is carried out;
C. And the master management node sends the metadata updating request to all the slave management nodes so that all the management nodes in the cross-data center storage architecture can update according to the metadata information to be updated.
Wherein, the step C specifically comprises:
c1, the master management node sends the request for updating the metadata to all the slave management nodes;
C2, the master management node receives first feedback information of all the slave management nodes, wherein the first feedback information comprises agreement update information;
And C3, when the quantity of the update consent information in the first feedback information received by the master management node is greater than a preset value, sending a command for executing updating of metadata information to all slave management nodes in the cross-data center storage architecture, so that all management nodes in the cross-data center storage architecture can update according to the metadata information to be updated.
After the step C3, the method may further include:
c4, when the quantity of the consent-to-update information in the first feedback information received by the master management node is greater than or equal to a preset value, the master management node feeds back a message that metadata information is updated successfully;
And C5, when the quantity of the consent-to-update information in the first feedback information received by the master management node is less than a preset value, the master management node feeds back a message that metadata information update fails.
wherein, still include: a method for selecting a master management node from all normally operating management nodes in a cross-data center storage architecture, wherein each piece of metadata information stored in any management node in the cross-data center storage architecture is allocated with a unique object ID number, the object ID number is increased along with updating of the metadata information, and for any management node in the cross-data center storage architecture, the management node is called a first management node, and the method comprises the following steps:
D1, setting the state of the first management node as a searching state;
D2, sending a request for setting the first management node as a master management node to all other management nodes in the cross-data center storage architecture;
D3, sending approved master management node information to all management nodes in the cross-data center storage architecture;
D4, receiving second feedback information of other management nodes, wherein the second feedback information comprises state information of the management nodes and master management node information approved by the management nodes, and the state information comprises searching state information, master management node state information and slave management node state information;
D5, when the received state information of a third management node in other management nodes is searching state information, comparing the size of the maximum transaction ID number in the first management node with the size of the maximum transaction ID number in the third management node, when the maximum transaction ID number of the first management node is larger than the maximum transaction ID number of the third management node, entering the step D6, otherwise, entering the step D7;
D6, changing the state information of the first management node into the state information of the master management node, and entering the step D3;
d7, calculating the number of management nodes approving the third management node as the master management node, when the number of the management nodes approving the third management node as the master management node is greater than or equal to a preset number, updating the state information of the first management node into slave management node state information, recording the master management node information as the third management node, ending, and when the number of the management nodes approving the third management node as the master management node is less than the preset number, entering the step D3;
D8, when all the received second feedback information of all the other management nodes is the slave management node state information, updating the state of the slave management node to be the master management node state, and ending;
and D9, when the received second feedback information of a second management node in the other management nodes is the state information of the main management node, updating the main management node information stored by the first management node to be the second management node, and ending.
the method for acquiring the metadata in the management nodes is further included, a peer-to-peer internet technology is adopted among all the management nodes in the cross-data center storage architecture, and the method comprises the following steps:
E1, any management node in the cross-data center storage architecture receives a metadata obtaining request from a client, wherein the metadata obtaining request comprises a metadata transaction ID number;
e2, the any management node feeds back the metadata with the thing ID number to the client.
a data processing device is applied to a cross-data center storage architecture, the cross-data center storage architecture comprises at least two management nodes, one management node of the at least two management nodes is used as a master management node, and the other management nodes are used as slave management nodes, the data processing device comprises:
A first receiving module, configured to receive, by any management node, a request for updating metadata, where the request for updating metadata includes metadata information to be updated;
The judging module is used for judging whether any management node is a main management node, if so, executing a first sending module, otherwise, sending the request for updating the metadata to the main management node, and executing the first sending module;
The first sending module is configured to send, by the master management node, the request for updating the metadata to all the slave management nodes, so that all the management nodes in the cross-data center storage architecture update according to the metadata information to be updated.
Wherein the first transmitting module comprises:
A sending unit, configured to send, by the master management node, the request for updating the metadata to all the slave management nodes;
a receiving unit, configured to receive, by the master management node, first feedback information of all the slave management nodes, where the first feedback information includes consent-update information;
And the execution unit is used for sending a command for executing updating of metadata information to all the slave management nodes in the cross-data center storage architecture when the quantity of the update agreeing information in the first feedback information received by the master management node is greater than a preset value, so that all the management nodes in the cross-data center storage architecture can be updated according to the metadata information to be updated.
Wherein, still include:
the first feedback unit is used for feeding back a message that metadata information is successfully updated when the quantity of the update agreeing information in the first feedback information received by the main management node is greater than or equal to a preset value;
A second feedback unit, configured to, when the quantity of the update consent information in the first feedback information received by the primary management node is smaller than a preset value, feed back a message indicating that metadata information update fails by the primary management node.
each piece of metadata information stored in any management node in the cross-data center storage architecture is allocated with a unique object ID number, and the object ID number is increased along with the updating of the metadata information, any management node in the cross-data center storage architecture further comprises a master management node selection device, and any management node is called a first management node, and the master management node selection device comprises:
the setting module is used for setting the state of the first management node as a searching state;
A second sending module, configured to send a request for setting the first management node as a master management node to all other management nodes in the cross-data center storage architecture;
the third sending module is used for sending approved main management node information to all management nodes in the cross-data center storage architecture;
The second receiving module is used for receiving second feedback information of other management nodes, wherein the second feedback information comprises state information of the management nodes and main management node information approved by the management nodes, and the state information comprises searching state information, main management node state information and slave management node state information;
The comparison module is used for comparing the size of the maximum object ID number in the first management node with the size of the maximum object ID number in the third management node when the received state information of the third management node in other management nodes is searching state information, and executing the modification module when the maximum object ID number of the first management node is larger than the maximum object ID number of the third management node, or executing the calculation module;
the modification module is configured to change the state information of the first management node into state information of a master management node, and return the state information to the third sending module;
The calculating module is configured to calculate the number of management nodes approving the third management node as a master management node, update the state information of the first management node to slave management node state information when the number of management nodes approving the third management node as a master management node is greater than or equal to a preset number, record master management node information as the third management node, end, and return to the third sending module when the number of management nodes approving the third management node as a master management node is smaller than the preset number;
the first updating module is used for updating the state of the first updating module to be the state of the master management node when all the received second feedback information of all the other management nodes is the state information of the slave management node, and ending;
and the second updating module is used for updating the main management node information stored by the first management node to be the second management node when the received second feedback information of the second management node in the other management nodes is the main management node state information, and ending the updating.
The data center-crossing storage architecture comprises a data center-crossing storage architecture, a management node and a metadata acquisition device, wherein the data center-crossing storage architecture further comprises a metadata acquisition device, a peer-to-peer internet technology is adopted among all management nodes in the data center-crossing storage architecture, and the metadata acquisition device comprises:
a third receiving module, configured to receive, by any management node in the cross-data center storage architecture, a metadata obtaining request from a client, where the metadata obtaining request includes a metadata transaction ID number;
A feedback module, configured to feed back, by the any management node, the metadata with the transaction ID number to the client.
the technical scheme has the following beneficial effects: when the metadata information needs to be updated, the master management node and the slave management nodes update the stored metadata information, so that the metadata information stored by the master management node and the slave management nodes is the same, and when any management node fails, the same metadata information is stored by other management nodes, so that idle time in the prior art cannot occur, and the availability of storage service in the cross-data center storage system is effectively improved.
Drawings
Fig. 1 is a schematic method flow diagram of an implementation manner of a data processing method according to an embodiment of the present invention;
Fig. 2 is a schematic method flow diagram illustrating an implementation manner of step C in a data processing method according to an embodiment of the present invention;
Fig. 3 is a schematic method flow diagram of an implementation manner of a method for selecting a master management node from all normally operating management nodes in the cross-data center storage architecture in the data processing method according to the embodiment of the present invention;
Fig. 4 is a schematic method flow diagram illustrating an implementation manner of a method for acquiring metadata in a management node in a data processing method according to an embodiment of the present invention;
Fig. 5 is a schematic structural diagram of an implementation manner of a data processing apparatus according to an embodiment of the present invention;
Fig. 6 is a schematic structural diagram of an implementation manner of a first sending module in a data processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an implementation manner of selecting a master management node device in a data processing device according to an embodiment of the present invention;
Fig. 8 is a schematic structural diagram of an implementation manner of an apparatus for acquiring metadata in a data processing apparatus according to an embodiment of the present invention.
Detailed Description
for the sake of reference and clarity, the descriptions, abbreviations or abbreviations of the technical terms used hereinafter are summarized as follows:
GFS: google File System, distributed File System;
HDFS (Hadoop distributed File System): a Hadoop Distributed File System;
CRC: cyclic Redundancy Check (crc), Cyclic Redundancy Check (crc);
NFS: network File System, Network File System;
TFS: the Taobao File System, an Internet-oriented distributed file system;
a Naneonide: name nodes.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
With the development of computer technology, the data volume in various fields tends to increase in a massive manner, the demand for data storage is higher and higher, and the requirement for data backup is also improved. For example, data applied to satellite remote sensing and the like is developing from PB level to EB level, and how to meet the requirement of EB level data storage becomes an urgent problem to be solved for a large data storage platform. Currently, multiple data centers are the main method for solving the big data challenge, and meet the requirement of data capacity expansion, for example, companies such as satellite application, Google, Facebook and the like establish multiple data centers globally. For the management inconvenience of data by multiple data centers, a cross-data center management architecture, such as the spider management system of Google, is proposed. Bottlenecks such as single management node faults of management nodes of a cross-data center management architecture cannot meet the requirement of EB-level data storage. At present, a distributed storage system and multi-data center storage are key technical approaches for meeting EB-level data storage requirements. The distributed storage system of the unified distributed storage system of the cross data center can effectively solve the problem of distributed storage and management of data in an application environment, and the system interconnects the distributed storage systems of different regions through a wide area network to provide EB-level storage architecture service.
in the current distributed environment, system failure, especially for cheap servers, is a relatively normal condition, which includes failures of software, hardware, etc., so a system failure recovery and guarantee method is required to ensure that the system operates normally. Typical distributed file systems GFS and HDFS employ specific fault-reporting measures, such as performing CRC checksum calculation on data blocks to ensure the integrity and correctness of the data blocks, which is a fault protection mechanism for data; in addition to a data fault protection mechanism, the HDFS adopts the ideas of NameNode metadata backup and NameNode active/standby switching to ensure the normal service of the system after the management node of the NameNode is down. The NameNode which normally operates keeps a plurality of copies through the NFS remote mount directory to synchronize local metadata information to a remote file server to ensure that a plurality of copies of the metadata information normally operate, but a service stopping time interval exists between the time when the main NameNode is down and the time when the main NameNode is switched to the backup NameNode to start. The new high-availability scheme of the HDFS ensures the high availability of the NameNodes in the HDFS by means of a third-party system ZooKeeper, wherein two NameNodes are used, one is in an active state (active) and the other is used for backup (standby), and the NameNodes of the standby do not provide services under the normal operation condition, so that the resources of the NameNodes of the system standby are not fully utilized. TFS is a distributed file system aiming at the network small picture mass storage requirement of E-commerce requirement, TFS adopts a heartbeat mechanism based on Linux to ensure the high availability scheme of name nodes, and the same method is realized by that the master NameNode sends heartbeat information to the slave NameNode, and once the heartbeat information of the master NameNode is not received in overtime, the slave NameNode takes over the service of the whole metadata management. A distributed database system HBase (distributed and column-oriented open source database) in a Hadoop ecosphere adopts the same high availability scheme for realizing HMASter based on ZooKeeper, a plurality of HMASers can be owned in the HBase, but only one HMASter can be used as a service node, and the HMASers compete to become a main HMASter at the same time.
the CRC code can only ensure the validity of the data block and cannot ensure the validity of the management node. And using NFS to mount and store a plurality of copies of the metadata, and then synchronizing the metadata to a local restart management node, wherein a service stop time interval exists in the cold standby switching concept. The novel hot standby switching idea of the master node realized based on the heartbeat mechanism and the ZooKeeper mechanism also has a service stopping time interval.
the service stop time interval is an idle time in the processes of synchronizing information on a remote file server to a normally-operated management node and replacing a main management node by a backup management node, and no management node stores metadata information updated or modified in the idle time.
please refer to fig. 1, which is a schematic method flow diagram of an implementation manner of a data processing method according to an embodiment of the present invention, where the data processing method is applied to a cross-data center storage architecture, where the cross-data center storage architecture includes at least two management nodes, and one of the at least two management nodes is used as a master management node and the other management nodes are used as slave management nodes, and the data processing method includes:
step A: any management node receives a request to update metadata.
the request to update the metadata includes metadata information to be updated. The above-mentioned update metadata information may be modification metadata information, addition metadata information, deletion metadata information, or the like. The metadata information is also called intermediary data and relay data, and is data describing data (data about data), mainly describing data attribute (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like.
Any management node in the cross-data center storage architecture can receive a request to update metadata, whether the management node is a master management node or a slave management node.
in order to implement that both the master management node and the slave management node can receive a request for updating metadata, a peer to peer structure between the master management node and the slave management node may be set, and the like.
The trigger of generating the "update metadata information request" across the data center storage architecture may be a case of detecting that different metadata information exists between the management nodes, or may be a case of generating one piece of metadata information, deleting the metadata information, or modifying the metadata information, for example, after new WORD is created, there is a set-up time of WORD, and the set-up time is newly added metadata information.
And B: and C, judging whether the management node is a main management node, if so, entering the step C, otherwise, sending the request for updating the metadata to the main management node, and entering the step C.
Any management node is a management node that receives the "metadata information to be updated".
The management node in the embodiment of the present invention may manage a plurality of distributed file system clusters, for example, HDFS clusters, and also include distributed database system clusters, for example, HBase clusters, so that the management node may support operation requests of the distributed file system and the distributed database system nosql. Because both the distributed file system cluster and the distributed database system cluster support a cluster-level capacity expansion mechanism, the cross-data center storage architecture in the embodiment of the invention can realize smooth increase and decrease of space dynamics, and can use a plurality of cluster data servers to store data at a back end.
And C: and the master management node sends a metadata updating request to all the slave management nodes so that all the management nodes in the cross-data center storage architecture can update according to the metadata information to be updated.
The master management node may update the metadata information at the same time as the slave management node, or may update the metadata information in a step earlier than the slave management node, which is not limited in this embodiment of the present invention.
The backup of a plurality of metadata is ensured by setting a plurality of management nodes, and the running reliability of a storage architecture in the cross-data is improved.
When the metadata information needs to be updated, the metadata information stored by the master management node and the slave management node is updated, so that the metadata information stored by the master management node and the slave management node is the same, and when any management node fails, the metadata information stored by other management nodes is the same, so that idle time in the prior art cannot occur.
referring to fig. 2, a method flow diagram of an implementation manner of step C in a data processing method according to an embodiment of the present invention is shown, where the method includes:
C1, the master management node sends the request for updating the metadata to all the slave management nodes.
c2, the master management node receives the first feedback information of all the slave management nodes.
the first feedback information includes consent-update information.
and C3, when the quantity of the update consent information in the first feedback information received by the master management node is greater than a preset value, sending a command for executing updating of metadata information to all slave management nodes in the cross-data center storage architecture, so that all management nodes in the cross-data center storage architecture can update according to the metadata information to be updated.
The specific method in the step C may be a paxos algorithm, which is an algorithm for ensuring consistency of multiple copies, and the metadata information synchronization on multiple management nodes is ensured by the method.
Generally, when receiving an update metadata request from a management node, the update is agreed, and it is determined that "the amount of agreed update information is greater than a preset value" takes time delay into consideration, because some slave management nodes need longer time to send first feedback information to a master management node, and in order to save time more quickly, as long as the agreed update request information received by the master management node exceeds the preset value, a command for executing update metadata information is sent to all slave management nodes in the cross-data center storage architecture.
Here, the judgment that the number of pieces of consent-update information is greater than the preset value also takes into consideration the situation that the consent information cannot be fed back in time due to a failure of the management node.
The preset value may be more than half of the total number of management nodes that normally operate in the storage architecture across the data center, for example, assuming that the total number of management nodes that normally operate in the storage architecture across the data center is 9, the preset value may be any one of 5, 6, 7, and 8.
The first feedback information may also include disapproval of the update information, for example, if the management node detects that it has updated the metadata information that needs to be updated this time, the request for updating the metadata information this time may be rejected.
After step C3, the method may further include: c4, when the quantity of the consent-to-update information in the first feedback information received by the master management node is greater than or equal to a preset value, the master management node feeds back a message that metadata information is updated successfully; and C5, when the quantity of the consent-to-update information in the first feedback information received by the master management node is less than a preset value, the master management node feeds back a message that the metadata information update fails.
referring to fig. 3, which is a schematic method flow diagram of an implementation manner of a method for selecting a master management node from all normally operating management nodes in the cross-data center storage architecture in a data processing method according to an embodiment of the present invention, each piece of metadata information stored in any one management node in the cross-data center storage architecture is assigned with a unique object ID number, and the object ID number is increased along with updating of the metadata information, and for any one management node in the cross-data center storage architecture, the any one management node is a first management node, where the method includes:
d1, setting the state of the first management node as a searching state.
when the management node performs metadata updating operation, the management node performs management of each metadata updating operation by adopting the ID characteristic of the object number. For example, when the object ID number of the current first management node is 5 and metadata information is newly added to the first management node, the object ID of the new metadata information is 6, and therefore, the object ID number of the metadata stored in the first management node is larger as the metadata is continuously updated in the first management node.
And D2, sending a request for setting the first management node as a master management node to all other management nodes in the cross-data center storage architecture.
At the beginning, the first management node recommends itself as the master management node.
D3, sending approved master management node information to all management nodes in the cross-data center storage architecture.
d4, receiving second feedback information of other management nodes, wherein the second feedback information comprises state information of the management nodes and main management node information approved by the management nodes.
The state information comprises searching state information, main management node state information and slave management node state information.
D5, when the received state information of the third management node in other management nodes is the searching state information, comparing the size of the maximum object ID number in the first management node with the size of the maximum object ID number in the third management node, when the maximum object ID number of the first management node is larger than the maximum object ID number of the third management node, entering the step D6, otherwise, entering the step D7.
the third management node is any management node except the first management node in the cross-data center storage architecture, and is referred to as the third management node for distinguishing from the first management node.
D6, changing the state information of the first management node into the state information of the master management node, and entering the step D3.
D7, calculating the number of the management nodes which approve the third management node as the master management node, when the number of the management nodes which approve the third management node as the master management node is larger than or equal to the preset number, updating the state information of the first management node into the state information of the slave management node, recording the information of the master management node as the third management node, ending, and when the number of the management nodes which approve the third management node as the master management node is smaller than the preset number, entering the step D3.
The preset number may be more than half of the total number of management nodes that normally operate in the cross-data center storage architecture, for example, assuming that the total number of management nodes that normally operate in the cross-data center storage architecture is 9, the preset number may be any one of 5, 6, 7, and 8.
And D8, when all the received second feedback information of all the other management nodes is the slave management node state information, updating the state of the slave management node to be the master management node state, and ending.
and D9, when the received second feedback information of a second management node in the other management nodes is the state information of the main management node, updating the main management node information stored by the first management node to be the second management node, and ending.
The subjects of the above steps D1 to D9 are all the first management nodes, i.e. applicable to any management node in the cross data center storage architecture.
the data processing method provided by the embodiment of the invention adopts the mode that the main management node proposes to reselect the main management node, thereby avoiding the livelock problem in the method for selecting the main management node. Livelock refers to a task or performer that is not blocked, resulting in repeated attempts, failures, attempts, and failures due to certain conditions not being met.
The management node as a backup in the prior art cannot fully exert the own value, namely, the master management node and the slave management node cannot serve the client at the same time. The invention provides the following methods for this purpose.
Referring to fig. 4, a method flow diagram of an implementation manner of a method for obtaining metadata in management nodes in a data processing method according to an embodiment of the present invention is shown, where peer-to-peer internet technology is used among all management nodes in a cross-data center storage architecture, and the method includes:
e1, any management node in the cross-data center storage architecture receives a metadata obtaining request from a client, wherein the metadata obtaining request comprises a metadata transaction ID number.
E2, the any management node feeds back the metadata with the thing ID number to the client.
The embodiment of the invention adopts the peer-to-peer internet technology, so that the request can be normally served by the master management node or the slave management nodes, one management node can be selected to access at the client, and the resources of a plurality of management nodes are fully utilized. Each management node can serve the request from the client, and if the request is a read request of metadata, the management node directly responds to a reply request; if the request is a metadata update request, the request is sent from the management node to the master management node.
The embodiment of the invention can be used in software, at the moment, the operating system can be a Linux system and runs on software for providing file IO service in a Linux cluster, such as a distributed file system like HDFS and a NoSQL distributed database system like HBase, and the HDFS distributed file system is provided with a plurality of DataNodes.
referring to fig. 5, a schematic structural diagram of an implementation manner of a data processing apparatus according to an embodiment of the present invention is shown, where the data processing apparatus is applied to a cross-data center storage architecture, where the cross-data center storage architecture includes at least two management nodes, and one of the at least two management nodes is a master management node, and the other management nodes are slave management nodes, and the data processing apparatus includes: a first receiving module 501, a judging module 502, and a first sending module 503, wherein:
a first receiving module 501, configured to receive, by any of the management nodes, a request for updating metadata.
The request to update the metadata includes metadata information to be updated. The above-mentioned update metadata information may be modification metadata information, addition metadata information, deletion metadata information, or the like. The metadata information is also called intermediary data and relay data, and is data describing data (data about data), mainly describing data attribute (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like.
the cross-data center storage architecture comprises at least two management nodes, specifically, more than or equal to 3 management nodes.
Any management node in the cross-data center storage architecture can receive a request to update metadata, whether the management node is a master management node or a slave management node.
in order to implement that both the master management node and the slave management node can receive a request for updating metadata, a peer-to-peer structure between the master management node and the slave management node may be set, and the like.
The trigger of generating the "update metadata information request" across the data center storage architecture may be a case of detecting that different metadata information exists between the management nodes, or may be a case of generating one piece of metadata information, deleting the metadata information, or modifying the metadata information, for example, after new WORD is created, there is a set-up time of WORD, and the set-up time is newly added metadata information.
A determining module 502, configured to determine whether any management node is a master management node, if so, execute a first sending module 503, and if not, send the request for updating the metadata to the master management node, and execute the first sending module 503.
Any management node is a management node that receives the "metadata information to be updated".
the management node in the embodiment of the invention can manage a plurality of distributed file system clusters such as HDFS clusters and also include distributed database system clusters such as HBase clusters, so that the management node can support the operation requests of the distributed file system and the distributed database system NoSQL. Because both the distributed file system cluster and the distributed database system cluster support a cluster-level capacity expansion mechanism, the cross-data center storage architecture in the embodiment of the invention can realize smooth increase and decrease of space dynamics, and can use a plurality of cluster data servers to store data at a back end.
The first sending module 503 is configured to send the request for updating the metadata to all the slave management nodes by the master management node, so that all the management nodes in the cross-data center storage architecture are updated according to the metadata information to be updated.
the master management node may update the metadata information at the same time as the slave management node, or may update the metadata information in a step earlier than the slave management node, which is not limited in this embodiment of the present invention.
the backup of a plurality of metadata is ensured by setting a plurality of management nodes, and the running reliability of a storage architecture in the cross-data is improved.
The embodiment of the invention provides a data processing device, when metadata information needs to be updated, a master management node and a slave management node both update the stored metadata information, so that the metadata information stored by the master management node and the slave management node are the same, and when any management node fails, because the metadata information stored by other management nodes is the same, idle time in the prior art can not occur.
referring to fig. 6, which is a schematic structural diagram of an implementation manner of a first sending module in a data processing apparatus according to an embodiment of the present invention, the first sending module may include: a sending unit 601, a receiving unit 602, and an executing unit 603, wherein:
a sending unit 601, configured to send the request for updating metadata to all the slave management nodes by the master management node.
A receiving unit 602, configured to receive, by the master management node, first feedback information of all the slave management nodes, where the first feedback information includes consent-update information.
an executing unit 603, configured to send, to all slave management nodes in the cross-data center storage architecture, a command for executing updating of metadata information when the quantity of the update approval information in the first feedback information received by the master management node is greater than a preset value, so that all management nodes in the cross-data center storage architecture are updated according to the metadata information to be updated.
The specific implementation manner of the first sending module may be a paxos algorithm, where the paxos algorithm is an algorithm for ensuring consistency of multiple copies, and the metadata information synchronization on multiple management nodes is ensured by the above method.
Generally, when receiving an update metadata request from a management node, the update is agreed, and it is determined that "the amount of agreed update information is greater than a preset value" takes time delay into consideration, because some slave management nodes need longer time to send first feedback information to a master management node, and in order to save time more quickly, as long as the agreed update request information received by the master management node exceeds the preset value, a command for executing update metadata information is sent to all slave management nodes in the cross-data center storage architecture.
here, the judgment that the number of pieces of consent-update information is greater than the preset value also takes into consideration the situation that the consent information cannot be fed back in time due to a failure of the management node.
The preset value may be more than half of the total number of management nodes that normally operate in the storage architecture across the data center, for example, assuming that the total number of management nodes that normally operate in the storage architecture across the data center is 9, the preset value may be any one of 5, 6, 7, and 8.
The first feedback information may also include disapproval of the update information, for example, if the management node detects that it has updated the metadata information that needs to be updated this time, the request for updating the metadata information this time may be rejected.
Any of the data processing apparatuses described above may further include: the first feedback unit is used for feeding back a message that metadata information is successfully updated when the quantity of the update agreeing information in the first feedback information received by a main management node is greater than or equal to a preset value; and the second feedback unit is used for feeding back a message of metadata information updating failure by the main management node when the quantity of the update agreeing information in the first feedback information received by the main management node is less than a preset value.
The client is the client that triggered this "update metadata request".
Referring to fig. 7, a schematic structural diagram of an implementation manner of selecting a master management node device in a data processing apparatus according to an embodiment of the present invention is shown, where metadata information stored in any management node in a cross-data center storage architecture is allocated with an object ID number, and the object ID number is increased along with updating of the metadata information, where any management node in the cross-data center storage architecture further includes a selected master management node device, and is referred to as a first management node, where the selected master management node device includes: a setting module 701, a second sending module 702, a third sending module 703, a second receiving module 704, a comparing module 705, a modifying module 706, a calculating module 707, a first updating module 708, and a second updating module 709, wherein:
A setting module 701, configured to set a state of the first management node as a search state.
When the management node performs metadata updating operation, the management node performs management of each metadata updating operation by adopting the ID characteristic of the object number. For example, when the object ID number of the current first management node is 5 and metadata information is newly added to the first management node, the object ID of the new metadata information is 6, and therefore, the object ID number of the metadata stored in the first management node is larger as the metadata is continuously updated in the first management node.
A second sending module 702, configured to send a request for setting the first management node as a master management node to all other management nodes in the cross-data center storage architecture.
a third sending module 703, configured to send approved master management node information to all management nodes in the cross-data center storage architecture.
at the beginning, the first management node recommends itself as the master management node.
A second receiving module 704, configured to receive second feedback information of other management nodes, where the second feedback information includes state information of the management node and information of a master management node approved by the management node.
the state information comprises searching state information, main management node state information and slave management node state information.
A comparing module 705, configured to compare the size of the largest object ID number in the first management node with the size of the largest object ID number stored in the third management node when the received state information of the third management node in the other management nodes is the search state information, execute the modifying module 706 when the largest object ID number of the first management node is greater than the largest object ID number in the third management node, and execute the calculating module 707 when the largest object ID number of the first management node is less than the largest object ID number of the third management node.
The third management node is any management node except the first management node in the cross-data center storage architecture, and is referred to as the third management node for distinguishing from the first management node. The modifying module 706 is configured to change the state information of the first management node into the state information of the master management node, and return the state information to the third sending module 703.
The calculating module 707 is configured to calculate the number of management nodes that approve the third management node as a master management node, update the state information of the first management node to slave management node state information when the number of management nodes that approve the third management node as a master management node is greater than or equal to a preset number, record that the master management node information is the third management node, end, and return to the third sending module 703 when the number of management nodes that approve the third management node as a master management node is less than the preset number.
The preset number may be more than half of the total number of management nodes that normally operate in the cross-data center storage architecture, for example, assuming that the total number of management nodes that normally operate in the cross-data center storage architecture is 9, the preset number may be any one of 5, 6, 7, and 8.
The first updating module 708 is configured to update the state of the first updating module to be the state of the master management node when all the received second feedback information of all the other management nodes is the state information of the slave management node, and then the process is finished.
A second updating module 709, configured to update, when the received second feedback information of a second management node in the other management nodes is primary management node state information, the primary management node information stored in the first management node to be the second management node, and end.
the data processing method provided by the embodiment of the invention adopts the mode that the main management node proposes to reselect the main management node, thereby avoiding the livelock problem in the method for selecting the main management node. Livelock refers to a task or performer that is not blocked, resulting in repeated attempts, failures, attempts, and failures due to certain conditions not being met.
the setting module 701, the second sending module 702, the third sending module 703, the second receiving module 704, the comparing module 705, the modifying module 706, the calculating module 707, the first updating module 708, and the second updating module 709 all belong to a first management node, that is, are applicable to any management node in a cross-data center storage architecture.
the data processing method provided by the embodiment of the invention adopts the mode that the main management node proposes to reselect the main management node, thereby avoiding the livelock problem in the method for selecting the main management node. Livelock refers to a task or performer that is not blocked, resulting in repeated attempts, failures, attempts, and failures due to certain conditions not being met.
the management node as a backup in the prior art cannot fully exert the own value, namely, the master management node and the slave management node cannot serve the client at the same time. To this end, the present invention provides the following means.
referring to fig. 8, a schematic structural diagram of an implementation manner of an apparatus for acquiring metadata in a data processing apparatus according to an embodiment of the present invention is shown, where peer-to-peer internet technology is used among all management nodes in a storage architecture across data centers, and the apparatus for acquiring metadata includes:
a third receiving module 801, configured to receive, by any management node in the cross-data center storage architecture, a metadata obtaining request from a client, where the metadata obtaining request includes a metadata transaction ID number.
a feedback module 802, configured to the any management node feed back the metadata with the transaction ID number to the client.
The embodiment of the invention adopts the peer-to-peer internet technology, so that the request can be normally served by the master management node or the slave management nodes, one management node can be selected to access at the client, and the resources of a plurality of management nodes are fully utilized. Each management node can serve the request from the client, and if the request is a read request of metadata, the management node directly responds to a reply request; if the request is a metadata updating request, a management node is sent to the main management node.
the embodiment of the invention can be used in software, at this time, the operating system can be a Linux system, and runs on software providing file IO service in a Linux cluster, such as a distributed file system like HDFS and a nosql distributed database system like HBase, and the HDFS distributed file system is configured with a plurality of datanodes.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The previous description of the provided embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features provided herein.
Claims (8)
1. a data processing method is applied to a cross-data center storage architecture, the cross-data center storage architecture comprises at least two management nodes, one management node of the at least two management nodes is used as a master management node, and the other management nodes are used as slave management nodes, and the data processing method comprises the following steps:
A. Any management node receives a request for updating metadata, wherein the request for updating the metadata comprises metadata information to be updated;
B. The management node judges whether the management node is a main management node, if so, the step C is carried out, otherwise, the request for updating the metadata is sent to the main management node, and the step C is carried out;
C. The master management node sends the metadata updating request to all the slave management nodes so that all the management nodes in the cross-data center storage architecture can update according to the metadata information to be updated;
a method for selecting a master management node from all normally operating management nodes in a cross-data center storage architecture, wherein each piece of metadata information stored in any management node in the cross-data center storage architecture is allocated with a unique object ID number, the object ID number is increased along with updating of the metadata information, and for any management node in the cross-data center storage architecture, the management node is called a first management node, and the method comprises the following steps:
d1, setting the state of the first management node as a searching state;
D2, sending a request for setting the first management node as a master management node to all other management nodes in the cross-data center storage architecture;
D3, sending approved master management node information to all management nodes in the cross-data center storage architecture;
d4, receiving second feedback information of other management nodes, wherein the second feedback information comprises state information of the management nodes and master management node information approved by the management nodes, and the state information comprises searching state information, master management node state information and slave management node state information;
D5, when the received state information of a third management node in other management nodes is searching state information, comparing the size of the maximum transaction ID number in the first management node with the size of the maximum transaction ID number in the third management node, when the maximum transaction ID number of the first management node is larger than the maximum transaction ID number of the third management node, entering the step D6, otherwise, entering the step D7;
D6, changing the state information of the first management node into the state information of the master management node, and entering the step D3;
d7, calculating the number of management nodes approving the third management node as the master management node, when the number of the management nodes approving the third management node as the master management node is greater than or equal to a preset number, updating the state information of the first management node into slave management node state information, recording the master management node information as the third management node, ending, and when the number of the management nodes approving the third management node as the master management node is less than the preset number, entering the step D3;
D8, when all the received second feedback information of all the other management nodes is the slave management node state information, updating the state of the slave management node to be the master management node state, and ending;
and D9, when the received second feedback information of a second management node in the other management nodes is the state information of the main management node, updating the main management node information stored by the first management node to be the second management node, and ending.
2. the data processing method according to claim 1, wherein the step C specifically comprises:
C1, the master management node sends the request for updating the metadata to all the slave management nodes;
c2, the master management node receives first feedback information of all the slave management nodes, wherein the first feedback information comprises agreement update information;
and C3, when the quantity of the update approval information in the first feedback information received by the master management node is greater than a preset value, sending a command for executing updating of metadata information to all slave management nodes in the cross-data center storage architecture, so that all management nodes in the cross-data center storage architecture can update according to the metadata information to be updated.
3. the data processing method according to claim 2, further comprising, after step C3:
c4, when the quantity of the update agreeing information in the first feedback information received by the master management node is greater than or equal to a preset value, the master management node feeds back a message that the metadata information is successfully updated;
C5, when the quantity of the update agreeing information in the first feedback information received by the master management node is less than a preset value, the master management node feeds back a message that the metadata information update fails.
4. the data processing method according to claim 1, 2 or 3, further comprising a method for obtaining metadata in management nodes, wherein peer-to-peer internet technology is adopted among all management nodes in the cross-data center storage architecture, and the method comprises:
E1, any management node in the cross-data center storage architecture receives a request for obtaining the updating metadata from a client, wherein the request for obtaining the updating metadata comprises a metadata object ID number;
e2, the any management node feeds back the metadata with the thing ID number to the client.
5. A data processing apparatus, applied to a cross-data center storage architecture, where the cross-data center storage architecture includes at least two management nodes, one of the at least two management nodes is a master management node, and the other management nodes are slave management nodes, the data processing apparatus includes:
A first receiving module, configured to receive, by any management node, a request for updating metadata, where the request for updating metadata includes metadata information to be updated;
the judging module is used for judging whether any management node is a main management node, if so, executing a first sending module, otherwise, sending the request for updating the metadata to the main management node, and executing the first sending module;
The first sending module is configured to send, by the master management node, the metadata updating request to all the slave management nodes, so that all the management nodes in the cross-data center storage architecture update according to the metadata information to be updated;
each piece of metadata information stored in any management node in the cross-data center storage architecture is distributed with a unique object ID number, the object ID number is increased along with the updating of the metadata information, any management node in the cross-data center storage architecture further comprises a main selection management node device, the main selection management node device is called a first management node, and the main selection management node device comprises:
the setting module is used for setting the state of the first management node as a searching state;
a second sending module, configured to send a request for setting the first management node as a master management node to all other management nodes in the cross-data center storage architecture;
The third sending module is used for sending approved main management node information to all management nodes in the cross-data center storage architecture;
the second receiving module is used for receiving second feedback information of other management nodes, wherein the second feedback information comprises state information of the management nodes and main management node information approved by the management nodes, and the state information comprises searching state information, main management node state information and slave management node state information;
The comparison module is used for comparing the size of the maximum object ID number in the first management node with the size of the maximum object ID number in the third management node when the received state information of the third management node in other management nodes is searching state information, and executing the modification module when the maximum object ID number of the first management node is larger than the maximum object ID number of the third management node, or executing the calculation module;
the modification module is configured to change the state information of the first management node into state information of a master management node, and return the state information to the third sending module;
The calculating module is configured to calculate the number of management nodes approving the third management node as a master management node, update the state information of the first management node to slave management node state information when the number of management nodes approving the third management node as a master management node is greater than or equal to a preset number, record master management node information as the third management node, end, and return to the third sending module when the number of management nodes approving the third management node as a master management node is smaller than the preset number;
The first updating module is used for updating the state of the first updating module to be the state of the master management node when all the received second feedback information of all the other management nodes is the state information of the slave management node, and ending;
and the second updating module is used for updating the main management node information stored by the first management node to be the second management node when the received second feedback information of the second management node in the other management nodes is the main management node state information, and ending the updating.
6. The data processing apparatus of claim 5, wherein the first sending module comprises:
A sending unit, configured to send, by the master management node, the request for updating the metadata to all the slave management nodes;
A receiving unit, configured to receive, by the master management node, first feedback information of all the slave management nodes, where the first feedback information includes consent-update information;
And the execution unit is used for sending a command for executing updating of metadata information to all the slave management nodes in the cross-data center storage architecture when the quantity of the update agreeing information in the first feedback information received by the master management node is greater than a preset value, so that all the management nodes in the cross-data center storage architecture can be updated according to the metadata information to be updated.
7. the data processing apparatus of claim 6, further comprising:
The first feedback unit is used for feeding back a message that metadata information is successfully updated when the quantity of the update approval information in the first feedback information received by the main management node is greater than or equal to a preset value;
a second feedback unit, configured to, when the quantity of the update approval information in the first feedback information received by the primary management node is smaller than a preset value, feed back a message indicating that metadata information update fails by the primary management node.
8. The data processing apparatus according to claim 5, 6 or 7, further comprising means for obtaining metadata, wherein peer-to-peer internet technology is adopted between all management nodes in the cross-data center storage architecture, and the means for obtaining metadata comprises:
a third receiving module, configured to receive, by any management node in the cross-data center storage architecture, a request for obtaining update metadata from a client, where the request for obtaining update metadata includes a metadata transaction ID number;
A feedback module, configured to feed back, by the any management node, the metadata with the transaction ID number to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510524829.6A CN105069152B (en) | 2015-08-25 | 2015-08-25 | data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510524829.6A CN105069152B (en) | 2015-08-25 | 2015-08-25 | data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105069152A CN105069152A (en) | 2015-11-18 |
| CN105069152B true CN105069152B (en) | 2019-12-13 |
Family
ID=54498521
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510524829.6A Active CN105069152B (en) | 2015-08-25 | 2015-08-25 | data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105069152B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106325768B (en) * | 2016-08-19 | 2019-08-13 | 华中科技大学 | A kind of two-shipper storage system and method |
| CN107277111B (en) * | 2017-05-19 | 2018-09-04 | 腾讯科技（深圳）有限公司 | A kind of company-data processing method, master node, slave node and cluster |
| CN107832138B (en) * | 2017-09-21 | 2021-09-14 | 南京邮电大学 | Method for realizing flattened high-availability namenode model |
| CN107786650B (en) * | 2017-10-20 | 2021-06-18 | 郑州云海信息技术有限公司 | Storage medium and management strategy synchronization method, device and system |
| CN109995554A (en) * | 2017-12-29 | 2019-07-09 | 中国移动通信集团吉林有限公司 | The control method and cloud dispatch control device of multi-stage data center active-standby switch |
| CN110633168A (en) * | 2018-06-22 | 2019-12-31 | 北京东土科技股份有限公司 | Data backup method and system for distributed storage system |
| CN111400112B (en) * | 2020-03-18 | 2021-04-13 | 深圳市腾讯计算机系统有限公司 | Writing method and device of storage system of distributed cluster and readable storage medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101730200A (en) * | 2008-10-13 | 2010-06-09 | 大唐移动通信设备有限公司 | Method, device and system for selecting target node, and operation, management and maintenance server |
| CN102122306A (en) * | 2011-03-28 | 2011-07-13 | 中国人民解放军国防科学技术大学 | Data processing method and distributed file system applying same |
| CN102523251A (en) * | 2011-11-25 | 2012-06-27 | 北京开拓天际科技有限公司 | Cloud storage architecture for processing mass data and cloud storage platform using the same |
| CN103139302B (en) * | 2013-02-07 | 2015-07-08 | 浙江大学 | Real-time copy scheduling method considering load balancing |
| CN103384266B (en) * | 2013-06-07 | 2016-12-28 | 曙光信息产业(北京)有限公司 | A kind of Parastor200 based on file-level real-time synchronization manages node high availability method |
| US20150081694A1 (en) * | 2013-09-18 | 2015-03-19 | International Business Machines Corporation | Multi-temporal widely distributed hardware and software transaction state and data state memory system |
| CN103944981B (en) * | 2014-04-14 | 2017-03-22 | 中国科学院计算技术研究所 | Cloud storage system and implement method based on erasure code technological improvement |
| CN104135539B (en) * | 2014-08-15 | 2018-03-16 | 华为技术有限公司 | Date storage method, SDN controllers and distributed network storage system |
-
2015
- 2015-08-25 CN CN201510524829.6A patent/CN105069152B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN105069152A (en) | 2015-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105069152B (en) | data processing method and device | |
| US11360854B2 (en) | Storage cluster configuration change method, storage cluster, and computer system | |
| US11379461B2 (en) | Multi-master architectures for distributed databases | |
| JP6382454B2 (en) | Distributed storage and replication system and method | |
| US9201742B2 (en) | Method and system of self-managing nodes of a distributed database cluster with a consensus algorithm | |
| US8301600B1 (en) | Failover recovery in a distributed data store | |
| US10922303B1 (en) | Early detection of corrupt data partition exports | |
| US20140108532A1 (en) | System and method for supporting guaranteed multi-point delivery in a distributed data grid | |
| EP2434729A2 (en) | Method for providing access to data items from a distributed storage system | |
| US9367261B2 (en) | Computer system, data management method and data management program | |
| US20180004777A1 (en) | Data distribution across nodes of a distributed database base system | |
| US9659078B2 (en) | System and method for supporting failover during synchronization between clusters in a distributed data grid | |
| US20120278429A1 (en) | Cluster system, synchronization controlling method, server, and synchronization controlling program | |
| CN113010496B (en) | Data migration method, device, equipment and storage medium | |
| CN112883119B (en) | Data synchronization method and device, computer equipment and computer readable storage medium | |
| CN112235405A (en) | Distributed storage system and data delivery method | |
| US10853892B2 (en) | Social networking relationships processing method, system, and storage medium | |
| CN105323271B (en) | Cloud computing system and processing method and device thereof | |
| CN109992447B (en) | Data copying method, device and storage medium | |
| CN116974983A (en) | Data processing method, device, computer readable medium and electronic equipment | |
| CN114130035A (en) | User matching method, device, equipment and storage medium | |
| CN109947593B (en) | Data disaster tolerance method, system, strategy arbitration device and storage medium | |
| CN116319623A (en) | Metadata processing method and device, electronic equipment and storage medium | |
| Meroufel et al. | Adaptive checkpointing with reliable storage in cloud environment | |
| CN115878269A (en) | Cluster migration method, related device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |