CN116226274A - Metadata synchronization method, distributed database system, and storage medium - Google Patents

Metadata synchronization method, distributed database system, and storage medium Download PDF

Info

Publication number
CN116226274A
CN116226274A CN202211705046.4A CN202211705046A CN116226274A CN 116226274 A CN116226274 A CN 116226274A CN 202211705046 A CN202211705046 A CN 202211705046A CN 116226274 A CN116226274 A CN 116226274A
Authority
CN
China
Prior art keywords
node
metadata
layer
layer metadata
pointer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211705046.4A
Other languages
Chinese (zh)
Inventor
于涛
俞新程
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jinzhuan Xinke Co Ltd
Original Assignee
Jinzhuan Xinke Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jinzhuan Xinke Co Ltd filed Critical Jinzhuan Xinke Co Ltd
Priority to CN202211705046.4A priority Critical patent/CN116226274A/en
Publication of CN116226274A publication Critical patent/CN116226274A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/217—Database tuning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2471—Distributed queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer And Data Communications (AREA)

Abstract

The invention discloses a metadata synchronization method, a distributed database system and a storage medium. The method is applied to a distributed database system, and the distributed database system comprises a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; and when receiving a metadata updating instruction sent by the service node, the first layer metadata node informs the second layer metadata node to modify and synchronize according to the matching relation. The scheme provided by the invention can accelerate the metadata synchronization speed and improve the performance of the distributed database system.

Description

Metadata synchronization method, distributed database system, and storage medium
Technical Field
The present invention relates to the field of database technologies, and in particular, to a metadata synchronization method, a distributed database system, and a storage medium.
Background
The database is the bottom core component of the financial business system. In a large-scale distributed database product, metaData contains a data dictionary, distribution rules, system configuration information and the like of each table, and records MetaData information such as how many storage nodes are under a distributed database cluster and corresponding to IP, ports and the like.
After receiving the SQL request, the distributed database accesses the metadata to acquire all cluster information, so that SQL analysis, routing and other works can be correctly performed. Almost every request from a client must query the relevant metadata information once. Metadata is uniformly served by a metadata management node MetaDataServer, abbreviated as MDS. When the system scale is large, the single-node metadata service cannot provide a service of high concurrency request. It is necessary to extend to multiple MDS distribution replica nodes.
When metadata is changed, a plurality of distributed replica nodes generate a data consistency problem, and the existing serialization synchronization logic can reduce performance indexes, so that the existing distributed database metadata synchronization process has defects.
Disclosure of Invention
In order to solve the technical problem that the system performance is low while strong consistency is met in the existing metadata synchronization process, the embodiment of the invention provides a metadata synchronization method, a distributed database system and a storage medium.
The technical scheme of the embodiment of the invention is realized as follows:
the embodiment of the invention provides a metadata synchronization method which is applied to a distributed database system, wherein the distributed database system comprises a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the method comprises the following steps: and when receiving a metadata updating instruction sent by the service node, the first layer metadata node informs the second layer metadata node to modify and synchronize according to the matching relation.
In an embodiment, the first layer metadata node includes a first layer primary node and a first layer backup node, and performs primary-backup switching when the first layer primary node fails.
In an embodiment, when receiving a metadata update instruction sent by a service node, the first layer metadata node notifies the second layer metadata node to modify and synchronize according to the matching relationship, including:
and inquiring the matching relation between the service node and the second-layer metadata node through the first-layer metadata node, modifying metadata information in the corresponding second-layer metadata node, constructing pointer routing information, and synchronizing metadata information in other second-layer metadata nodes through the pointer routing information.
In an embodiment, the constructing pointer routing information, through which metadata information in other second layer metadata nodes is synchronized, includes:
the starting process asynchronously modifies the pointer routing information into specific data;
and synchronizing the specific data according to the pointer routing information when synchronizing the metadata information in other second-layer metadata nodes.
In an embodiment, when receiving a metadata update instruction sent by a service node, the first layer metadata node notifies the second layer metadata node to modify and synchronize according to the matching relationship, including:
in the preparation stage, the first layer metadata node sends a transaction preprocessing request to a second layer metadata node corresponding to the service node; the corresponding second-layer metadata node is newly added with a copy node, a table construction statement of the newly added copy node is updated, and the second-layer metadata node is not submitted after execution, and a success response is sent to the first-layer metadata node; updating pointer routing information by other second-layer metadata nodes, pointing to original copy nodes in the corresponding second-layer metadata nodes, adding copy nodes by other second-layer metadata nodes, and synchronously replacing the added copy node information with pointer routing information; the other second-layer metadata nodes do not submit transactions after the execution is completed, and successful responses are sent to the coordinator;
and in the commit stage, after receiving successful responses returned by all the second-layer metadata nodes, the first-layer metadata node notifies all the second-layer metadata nodes to commit the transaction.
In an embodiment, the updating the pointer routing information includes:
acquiring a pointer routing address of a copy;
locating the latest modified copy node according to the content of the pointer;
and reading the duplicate node data, and assigning the read specific data to a pointer.
In an embodiment, in the commit phase, after the first layer metadata node receives successful responses returned by all second layer metadata nodes, notifying all second layer metadata nodes to commit the transaction, including:
after receiving the request for submitting the transaction, the corresponding second-layer metadata node covers the content of the original copy node, completes updating of the table-building statement and completes transaction submission;
after the change of the copy information pointed by the pointer route is monitored, an asynchronous task is started to replace data to modify the pointer route information;
and the newly-added copy information of other second-layer metadata nodes is synchronously changed, the content of the original copy node is covered, the updating of the table-building statement is completed, and the transaction submission is completed.
In an embodiment, after the completion transaction commits, the method further comprises:
releasing transaction resources occupied during the execution of the entire transaction.
The embodiment of the invention also provides a distributed database system, which comprises a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the distributed database system performs the steps of any of the methods described above.
The embodiment of the invention also provides a storage medium, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of any one of the methods are realized.
The metadata synchronization method, the distributed database system and the storage medium provided by the embodiment of the invention are applied to the distributed database system, wherein the distributed database system comprises a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the method comprises the following steps: and when receiving a metadata updating instruction sent by the service node, the first layer metadata node informs the second layer metadata node to modify and synchronize according to the matching relation. The scheme provided by the invention can accelerate the metadata synchronization speed and improve the performance of the distributed database system.
Drawings
FIG. 1 is a schematic diagram of a distributed database system according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a synchronization preparation stage according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of background synchronous routing information according to an embodiment of the present invention;
FIG. 4 is a flowchart of background replacement of synchronization metadata according to an embodiment of the present invention;
FIG. 5 is a flow chart of a synchronous commit phase according to an embodiment of the present invention;
FIG. 6 is a flow chart of adding mds node synchronization data in real time according to an embodiment of the present invention;
fig. 7 is an internal structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
The embodiment of the invention provides a metadata synchronization method which is applied to a distributed database system shown in figure 1, wherein the distributed database system comprises a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the method comprises the following steps:
and when receiving a metadata updating instruction sent by the service node, the first layer metadata node informs the second layer metadata node to modify and synchronize according to the matching relation.
The metadata nodes in this embodiment are divided into 2 layers, a first layer metadata node and a second layer metadata node. The first layer metadata node can adopt an HA high availability mode, divide the primary and standby nodes and save the matching relation between the service node and the MDS node. The second layer metadata node stores specific metadata information. In the second layer metadata node, copies of metadata required for a service may be put together according to the service type.
In an embodiment, the first layer metadata node includes a first layer primary node and a first layer backup node, and performs primary-backup switching when the first layer primary node fails.
The first layer metadata node is divided into a first layer main node and a first layer standby node, when the first layer main node fails, main and standby switching is carried out, and therefore, the situation that all participants in an uncommitted state are blocked due to the failure of a coordinator can be prevented.
In an embodiment, when receiving a metadata update instruction sent by a service node, the first layer metadata node notifies the second layer metadata node to modify and synchronize according to the matching relationship, including:
and inquiring the matching relation between the service node and the second-layer metadata node through the first-layer metadata node, modifying metadata information in the corresponding second-layer metadata node, constructing pointer routing information, and synchronizing metadata information in other second-layer metadata nodes through the pointer routing information.
In one embodiment, the constructing pointer routing information, synchronizing metadata information in other second-layer metadata nodes through the pointer routing information includes:
the starting process asynchronously modifies the pointer routing information into specific data;
and synchronizing the specific data according to the pointer routing information when synchronizing the metadata information in other second-layer metadata nodes.
In this embodiment, when a certain client service node needs to update metadata, the first layer of master node metadata node coordinator and the second layer of metadata node as participants perform metadata synchronization. Specifically, the data in the second-layer metadata node of the current service connection is modified first, pointer routing information is then constructed, and other second-layer metadata nodes are synchronized through pointer routing. The background then initiates a special process to asynchronously modify the routing information to specific data. By using the pointer route information to identify the latest modification, specific data is not required to be synchronized when the copies are synchronized, and only one piece of route information is required to be synchronized according to the pointer route, so that the synchronization speed of metadata can be increased.
Two specific examples will be described in detail below.
First embodiment
In an embodiment, when receiving a metadata update instruction sent by a service node, the first layer metadata node notifies the second layer metadata node to modify and synchronize according to the matching relationship, including:
in the preparation stage, the first layer metadata node sends a transaction preprocessing request to a second layer metadata node corresponding to the service node; the corresponding second-layer metadata node is newly added with a copy node, a table construction statement of the newly added copy node is updated, and the second-layer metadata node is not submitted after execution, and a success response is sent to the first-layer metadata node; updating pointer routing information by other second-layer metadata nodes, pointing to original copy nodes in the corresponding second-layer metadata nodes, adding copy nodes by other second-layer metadata nodes, and synchronously replacing the added copy node information with pointer routing information; the other second-layer metadata nodes do not submit transactions after the execution is completed, and successful responses are sent to the coordinator;
and in the commit stage, after receiving successful responses returned by all the second-layer metadata nodes, the first-layer metadata node notifies all the second-layer metadata nodes to commit the transaction.
The present embodiment employs a two-stage protocol for metadata synchronization.
In this embodiment, updating pointer routing information includes: acquiring a pointer routing address of a copy; locating the latest modified copy node according to the content of the pointer; and reading the duplicate node data, and assigning the read specific data to a pointer.
In actual operation, in the commit stage, after receiving successful responses returned by all the second-layer metadata nodes, the first-layer metadata node notifies all the second-layer metadata nodes to commit the transaction, including: after receiving the request for submitting the transaction, the corresponding second-layer metadata node covers the content of the original copy node, completes updating of the table-building statement and completes transaction submission; after the change of the copy information pointed by the pointer route is monitored, an asynchronous task is started to replace data to modify the pointer route information; and the newly-added copy information of other second-layer metadata nodes is synchronously changed, the content of the original copy node is covered, the updating of the table-building statement is completed, and the transaction submission is completed.
In this embodiment, after the transaction commit is completed, the transaction resources occupied during the execution of the entire transaction are released.
Referring to fig. 2, in this implementation, the implementation steps for large-scale synchronization of metadata can be further divided into:
(1) Step one, layering.
The metadata is divided into 2 layers, and the first layer stores A, B matching relations between two types of services and MDS nodes. The second layer is divided according to the matching relation stored in the first layer.
In this embodiment, as shown in fig. 1, an external requirement is transmitted to a first layer, a table establishment statement is required to be modified, and a first layer metadata node notifies a second layer copy node to carry out modification synchronization according to a stored matching relationship.
(2) Step two, the preparation stage of the two-stage protocol.
The first tier acts as a coordinator to send a transaction preprocessing request to the participant nodes of the second tier. A1 in FIG. 2 first executes a local transaction, adds a copy node, updates the table-building statement of the newly added copy node, does not commit after execution, and sends a Yes response to the coordinator.
For other nodes, the pointer route is updated to point to the original replica node in a 1. Other nodes route the synchronous copy information according to the pointers.
When synchronizing through pointer routing, if each copy node gets one time of data according to the pointing direction of the pointer, the synchronization efficiency is lower because the data volume of the table-building statement is large. In order to realize efficient and large-scale synchronization, a special process is started in the background, and the constructed routing information is changed into specific copy data pointed by the special process.
In this embodiment, the background process for replacing the pointer routing information changes the pointer routing information to the original copy node information of a 1. And the background process for replacing the synchronization metadata adds a copy node to other nodes and synchronously replaces the newly added node information with pointer routing information. The other nodes do not commit the transaction after execution, and send a Yes response to the coordinator.
And generating the asynchronous route replacement task to replace route information in the background as specific data information, so that the data access speed of the later nodes is improved, and indirect route access is not needed. And the data access with strong consistency is realized through the final consistency of the routing information.
The workflow of the background process for replacing pointer routing information can be seen in fig. 3, the pointer routing address of the copy is obtained, the latest modified copy node is located according to the content of the pointer, the node data is read, the read specific data is assigned to the pointer, and finally, the response is sent to the background process for replacing synchronous metadata, so that the process can be informed of starting metadata synchronization.
As shown in fig. 4, in the preparation stage, a background process for replacing the synchronization metadata first acquires pointer routing information, performs a new copy node operation on other nodes, and then assigns data in the pointer routing to all the new nodes.
(3) Step three, the submitting stage of the two-stage protocol.
After the first layer coordinator obtains the 'Yes' response returned by all participants, all participants are notified to submit the transaction.
In this embodiment, as shown in fig. 4, after a1 receives the request for submitting a transaction, the new copy node covers the content of the original copy node, and updates the table-building statement. a1, after completing the commit, releasing the resources occupied during the execution of the whole transaction.
after a1 is submitted, the contents of the copy nodes pointed by the pointer route are changed. And the background process monitors copy information pointed by the pointer route, and after the copy information is changed, the asynchronous task is started to replace data to modify the pointer route information. After the route information is changed, as shown in fig. 4, the newly added copy information of other nodes is also changed synchronously, and then the content of the original copy node is covered. After the execution is completed, other nodes carry out transaction submission, and finally transaction resources occupied during the whole transaction execution are released.
(4) Failure mode
The coordinator goes down. Since the first tier coordinator is down, it is unable to send commit requests, and all participants in the operational but uncommitted state are involved in a blocking situation. The HA high availability mode is introduced at the first layer. As shown in fig. 2, one node in the first layer is in a main state, and performs a distributing operation on the second layer; the other node is in a backup state and does not carry out distributing and responding operations. And synchronizing state information between the main node and the standby node, and switching the main node and the standby node when the main node goes down. Other failure modes all adopt the exception handling mode of the two-stage protocol.
Second embodiment
Referring to fig. 6, the method of synchronizing data when mds nodes are added in real time in the present embodiment is further analyzed as follows:
in this embodiment, the external demand is transmitted to the first layer node, and the master node notifies the second layer of adding a new service type new. Taking example new1 as an example, all nodes synchronize through pointer routing.
When synchronizing the user information nodes of new1 service, in the preparation stage, the first layer sends a request for preprocessing the transmission transaction to the second layer, and after receiving the request, the instance new1 starts user information synchronization, and constructs a pointer route to point to the user information nodes in b 1. The background starts a process for synchronizing the routing information, and replaces the routing information with the node information in b 1. After the process is finished, the background restarts the process of replacing the synchronous metadata, a copy node is newly added to new1, and the routing information is assigned to the newly added node. After the data of the newly added nodes are successfully synchronized, a Yes response is returned to the coordinator, and the coordinator sends a transaction submitting request to the second layer and enters a submitting stage. After the new1 node receives the request, the new copy information covers the original copy node information, and then the transaction is submitted. And after the user information is successfully synchronized, releasing transaction resources occupied during the whole transaction execution period.
When synchronizing the list-building statement of new1 service, in the preparation stage, the first layer sends a transaction preprocessing request to the second layer, and new1 starts to synchronize the list-building statement after receiving the request. Pointer routing points to the statement-building statement node in a1, at which point the duplicate node of a1 has just entered the commit phase and has not yet performed the commit of the transaction. After new1 adds the copy node, the metadata process is replaced through the background synchronous route information process and the background synchronous replacement, so that the synchronization of new1 added copy node information and the list-building sentence information of the a1 original copy node is realized. The preparation phase ends and new1 node enters the commit phase, at which point a1 performs commit transaction completion. The background synchronous route information process monitors that node data pointed by the pointer route is updated, and synchronously updates route information. And starting a background synchronous replacement metadata process, synchronously updating the table-building statement in new1, and covering the original node information with the newly added node information to complete the table-building statement synchronization. After the transaction is submitted, releasing transaction resources occupied during the whole transaction execution period, and establishing the table statement synchronously is successful.
According to the embodiment, metadata can be efficiently synchronized in a distributed database on a large scale, and the performance problems caused by the existing serialization synchronization logic are solved by adopting layering, two-stage protocols, pointer routing and HA high availability modes. Namely, the embodiment of the invention can avoid the high concurrency problem of directly modifying the single-point metadata by layering. In addition, by routing information tagging, the update time of the serialization synchronization logic can be greatly reduced.
The metadata synchronization method provided by the embodiment of the invention is applied to a distributed database system, wherein the distributed database system comprises a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the method comprises the following steps: and when receiving a metadata updating instruction sent by the service node, the first layer metadata node informs the second layer metadata node to modify and synchronize according to the matching relation. The scheme provided by the invention can accelerate the metadata synchronization speed and improve the performance of the distributed database system.
In order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides a distributed database system, where the distributed database system includes a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the distributed database system performs the steps of the method of any of the above.
To implement the method of the embodiments of the present invention, the embodiments of the present invention also provide a computer program product comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps of the method described above.
Based on the hardware implementation of the program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). In particular, in one embodiment, the computer device may be a terminal, the internal structure of which may be as shown in fig. 7. The computer apparatus includes a processor a01, a network interface a02, a display screen a04, an input device a05, and a memory (not shown in the figure) which are connected through a system bus. Wherein the processor a01 of the computer device is adapted to provide computing and control capabilities. The memory of the computer device includes an internal memory a03 and a nonvolatile storage medium a06. The nonvolatile storage medium a06 stores an operating system B01 and a computer program B02. The internal memory a03 provides an environment for the operation of the operating system B01 and the computer program B02 in the nonvolatile storage medium a06. The network interface a02 of the computer device is used for communication with an external terminal through a network connection. Which when executed by a processor a01, performs the method of any of the embodiments described above. The display screen a04 of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device a05 of the computer device may be a touch layer covered on the display screen, or may be a key, a track ball or a touch pad arranged on a casing of the computer device, or may be an external keyboard, a touch pad or a mouse.
It will be appreciated by those skilled in the art that the structure shown in fig. 7 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
The device provided by the embodiment of the invention comprises a processor, a memory and a program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the method of any one of the embodiments.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash memory (flashRAM). Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transshipment) such as modulated data signals and carrier waves.
It will be appreciated that the memory of embodiments of the invention may be either volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. Wherein the nonvolatile Memory may be Read Only Memory (ROM), programmable Read Only Memory (PROM, programmable Read-Only Memory), erasable programmable Read Only Memory (EPROM, erasable Programmable Read-Only Memory), electrically erasable programmable Read Only Memory (EEPROM, electrically Erasable Programmable Read-Only Memory), magnetic random access Memory (FRAM, ferromagnetic random access Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk Read Only Memory (CD-ROM, compact Disc Read-Only Memory); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be random access memory (RAM, random Access Memory), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (SRAM, static Random Access Memory), synchronous static random access memory (SSRAM, synchronous Static Random Access Memory), dynamic random access memory (DRAM, dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, synchronous Dynamic Random Access Memory), double data rate synchronous dynamic random access memory (ddr SDRAM, double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random access memory (ESDRAM, enhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM, syncLink Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, direct Rambus Random Access Memory). The memory described by embodiments of the present invention is intended to comprise, without being limited to, these and any other suitable types of memory.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims (10)

1. A metadata synchronization method, which is characterized by being applied to a distributed database system, wherein the distributed database system comprises a first layer metadata node and a second layer metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the method comprises the following steps:
and when receiving a metadata updating instruction sent by the service node, the first layer metadata node informs the second layer metadata node to modify and synchronize according to the matching relation.
2. The method of claim 1, wherein the first layer metadata node comprises a first layer master node and a first layer backup node, and wherein a master-backup switch is performed in the event of a failure of the first layer master node.
3. The method according to claim 1, wherein said notifying, by the first layer metadata node, the second layer metadata node to modify and synchronize according to the matching relationship when receiving a metadata update instruction sent by a service node, includes:
and inquiring the matching relation between the service node and the second-layer metadata node through the first-layer metadata node, modifying metadata information in the corresponding second-layer metadata node, constructing pointer routing information, and synchronizing metadata information in other second-layer metadata nodes through the pointer routing information.
4. A method according to claim 3, wherein said constructing pointer routing information by which metadata information in other second tier metadata nodes is synchronized comprises:
the starting process asynchronously modifies the pointer routing information into specific data;
and synchronizing the specific data according to the pointer routing information when synchronizing the metadata information in other second-layer metadata nodes.
5. The method according to claim 1, wherein said notifying, by the first layer metadata node, the second layer metadata node to modify and synchronize according to the matching relationship when receiving a metadata update instruction sent by a service node, includes:
in the preparation stage, the first layer metadata node sends a transaction preprocessing request to a second layer metadata node corresponding to the service node; the corresponding second-layer metadata node is newly added with a copy node, a table construction statement of the newly added copy node is updated, and the second-layer metadata node is not submitted after execution, and a success response is sent to the first-layer metadata node; updating pointer routing information by other second-layer metadata nodes, pointing to original copy nodes in the corresponding second-layer metadata nodes, adding copy nodes by other second-layer metadata nodes, and synchronously replacing the added copy node information with pointer routing information; the other second-layer metadata nodes do not submit transactions after the execution is completed, and successful responses are sent to the coordinator;
and in the commit stage, after receiving successful responses returned by all the second-layer metadata nodes, the first-layer metadata node notifies all the second-layer metadata nodes to commit the transaction.
6. The method of claim 5, wherein updating pointer routing information comprises:
acquiring a pointer routing address of a copy;
locating the latest modified copy node according to the content of the pointer;
and reading the duplicate node data, and assigning the read specific data to a pointer.
7. The method of claim 5, wherein, in the commit phase, after the first layer metadata node receives successful responses returned by all second layer metadata nodes, notifying all second layer metadata nodes to commit the transaction, comprising:
after receiving the request for submitting the transaction, the corresponding second-layer metadata node covers the content of the original copy node, completes updating of the table-building statement and completes transaction submission;
after the change of the copy information pointed by the pointer route is monitored, an asynchronous task is started to replace data to modify the pointer route information;
and the newly-added copy information of other second-layer metadata nodes is synchronously changed, the content of the original copy node is covered, the updating of the table-building statement is completed, and the transaction submission is completed.
8. The method of claim 7, wherein after the completion of the transaction commit, the method further comprises:
releasing transaction resources occupied during the execution of the entire transaction.
9. A distributed database system, the distributed database system comprising a first tier metadata node and a second tier metadata node; the first layer metadata node stores the matching relation between the service node and the second layer metadata node; the second layer metadata node stores metadata information; the distributed database system performs the steps of the method of any one of claims 1 to 8.
10. A storage medium having a computer program stored therein, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 8.
CN202211705046.4A 2022-12-28 2022-12-28 Metadata synchronization method, distributed database system, and storage medium Pending CN116226274A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211705046.4A CN116226274A (en) 2022-12-28 2022-12-28 Metadata synchronization method, distributed database system, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211705046.4A CN116226274A (en) 2022-12-28 2022-12-28 Metadata synchronization method, distributed database system, and storage medium

Publications (1)

Publication Number Publication Date
CN116226274A true CN116226274A (en) 2023-06-06

Family

ID=86586360

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211705046.4A Pending CN116226274A (en) 2022-12-28 2022-12-28 Metadata synchronization method, distributed database system, and storage medium

Country Status (1)

Country Link
CN (1) CN116226274A (en)

Similar Documents

Publication Publication Date Title
US8392482B1 (en) Versioning of database partition maps
US6466965B1 (en) Centralized affinity maintenance in a workload managed client/server data processing system
CN111143382B (en) Data processing method, system and computer readable storage medium
CN111881223B (en) Data management method, device, system and storage medium
US8972695B2 (en) Automatic allocation of data replicas
US8402119B2 (en) Real-load tuning of database applications
CN107787490A (en) Function is directly connected in distributed data base grid
CN102262674A (en) Data synchronization method and system for distributed memory database
CN112162846B (en) Transaction processing method, device and computer readable storage medium
CN110019251A (en) A kind of data processing system, method and apparatus
CN115114374B (en) Transaction execution method and device, computing equipment and storage medium
CN105550342B (en) A kind of data processing method of the distributed data base of all-transparent
US11704327B2 (en) Querying distributed databases
Yang From Google file system to omega: a decade of advancement in big data management at Google
CN111414356A (en) Data storage method and device, non-relational database system and storage medium
CN114661690A (en) Multi-version concurrency control and log clearing method, node, equipment and medium
CN116701413A (en) Main data processing method and device
CN113806309B (en) Metadata deleting method, system, terminal and storage medium based on distributed lock
Pankowski Consistency and availability of Data in replicated NoSQL databases
US11727022B2 (en) Generating a global delta in distributed databases
US10331627B2 (en) Method and system for unified technological stack management for relational databases
CN116226274A (en) Metadata synchronization method, distributed database system, and storage medium
CN113761075A (en) Method, device, equipment and computer readable medium for switching databases
CN113448775A (en) Multi-source heterogeneous data backup method and device
CN113761053A (en) Data query method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination