CN116360920A - Stream data management method, database cluster, device and equipment - Google Patents
Stream data management method, database cluster, device and equipment Download PDFInfo
- Publication number
- CN116360920A CN116360920A CN202310229323.7A CN202310229323A CN116360920A CN 116360920 A CN116360920 A CN 116360920A CN 202310229323 A CN202310229323 A CN 202310229323A CN 116360920 A CN116360920 A CN 116360920A
- Authority
- CN
- China
- Prior art keywords
- node
- stream data
- database
- data management
- slave
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013523 data management Methods 0.000 title claims abstract description 210
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000009826 distribution Methods 0.000 claims abstract description 18
- 238000007726 management method Methods 0.000 claims description 30
- 230000004044 response Effects 0.000 claims description 25
- 230000015654 memory Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 13
- 238000012423 maintenance Methods 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 7
- 230000008878 coupling Effects 0.000 abstract description 7
- 238000010168 coupling process Methods 0.000 abstract description 7
- 238000005859 coupling reaction Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 17
- 230000006870 function Effects 0.000 description 14
- 230000002159 abnormal effect Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 4
- 241000412611 Consul Species 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000010076 replication Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000002253 acid Substances 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45587—Isolation or security of virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a stream data management method, a database cluster, a device and equipment, which are used for solving the technical problem that a strong coupling relation between master nodes and slave nodes cannot be isolated in the prior art. According to the database types of the master node and the slave node, adjusting a database protocol supported by the stream data management device; the method comprises the steps of receiving and storing stream data from a master node by using a database protocol of the master node, and then sending the stream data to a slave node through the database protocol of the slave node according to a distribution object of the stream data.
Description
Technical Field
The present disclosure relates to the field of database technologies, and in particular, to a method, a database cluster, a device, and equipment for managing stream data.
Background
The open source databases used in the current market, such as the traditional relational database Mysql, postgresql, all meet the ACID requirement, have comprehensive and powerful functions, can meet the requirements of various production environments, support secondary development, and have high market share. In order to achieve the purpose of high availability of the databases, each database provides a bottom layer support for the upper layer to realize a master-slave architecture.
Meanwhile, the relational database cluster is usually composed of a master node and slave nodes, the master node and the slave nodes are in a strong coupling relationship, and when the master node needs to deploy a plurality of slave nodes, data needs to be synchronized in the plurality of slave nodes at the same time, so that the pressure of the master node is increased; and when the master node fails, the slave node is easily affected, so that the database cannot normally operate.
In view of this, how to isolate the strong coupling relationship between the master node and the slave node is a technical problem to be solved.
Disclosure of Invention
The embodiment of the application provides a stream data management method, a database cluster, a device and equipment, which are used for solving the technical problem that a strong coupling relation between master nodes and slave nodes cannot be isolated in the prior art.
The first aspect of the present application provides a stream data management method, applied to a stream data management device in a relational database cluster, where the stream data management device connects a master node and a slave node in the relational database cluster, and the master node and the slave node exchange data through the stream data management device, the method includes:
according to the database types of the master node and the slave node, adjusting a database protocol supported by the stream data management device;
And receiving and storing stream data from the master node by using the database protocol of the master node, and then sending the stream data to the slave node through the database protocol of the slave node according to the distribution object of the stream data.
Optionally, according to the database types to which the master node and the slave node belong, adjusting a database protocol supported by the stream data management device includes:
establishing a virtual slave node to connect with the master node, acquiring a database protocol to which the master node belongs, and adding the database protocol to a database protocol supported by the stream data management device;
and connecting the slave node as a master node, acquiring a database protocol to which the slave node belongs, and adding the database protocol to the database protocol supported by the stream data management device.
Optionally, after adjusting the database protocol supported by the stream data management device, the method includes:
sending heartbeat packets to all the master nodes and slave nodes by using the adjusted database protocol, receiving the state information returned by each node and updating the state information into a database cluster information management table;
judging whether a fault node exists according to the state information, and if so, marking the fault node in a database cluster information management table.
Optionally, after marking the failed node in the database cluster information management table, the method includes:
if the marked fault node is a master node, shielding the fault node out of the database cluster, and re-registering the fault node into the database cluster after the fault node is repaired and online;
and if the marked fault node is a slave node, acquiring data in the slave node from a master node, and synchronizing the data into the fault node so as to restore the fault node to be a normal node.
In a second aspect, the present application provides a database cluster, comprising:
the master node is used for issuing data and configuring slave nodes in the database cluster;
a slave node for receiving and storing data;
stream data management means connecting all of said master nodes and said slave nodes and performing the method according to any of the first aspects.
Optionally, the stream data management device is configured to:
when the slave node enters a maintenance state, temporarily storing the data sent to the slave node into the stream data management device, and sending a response message to the master node, wherein the response message indicates that the slave node has successfully received the data; and sending the data temporarily stored in the stream data management device to the slave node until the maintenance of the slave node is finished.
Optionally, the stream data management device is configured to:
storing a database cluster information management table, and setting an IP address for the master node according to the database cluster information management table; the database cluster information management table comprises state information of all nodes.
In a third aspect, the present application provides a stream data management apparatus, including:
the protocol synchronization module is used for adjusting a database protocol supported by the stream data management device according to the database types of the master node and the slave node;
and the data distribution module is used for receiving and storing the stream data from the master node by using the database protocol of the master node, and then sending the stream data to the slave node through the database protocol of the slave node according to the distribution object of the stream data.
Optionally, the protocol synchronization module is configured to:
establishing a virtual slave node, and connecting the master node by using a database protocol to which the master node belongs;
as a master node, the slave node is connected using a database protocol to which the slave node belongs.
Optionally, the protocol synchronization module is configured to:
sending heartbeat packets to all the master nodes and slave nodes, receiving state information returned by each node and updating the state information into a database cluster information management table;
Judging whether a fault node exists according to the state information, and if so, marking the fault node in a cluster information management table.
Optionally, the protocol synchronization module is configured to:
if the fault node is a master node, shielding the fault node from the database cluster, and re-registering the fault node into the database cluster after the fault node is repaired and online;
and if the fault node is a slave node, acquiring data in the slave node from a master node, and synchronizing the data in the slave node to the fault node so as to restore the fault node to be a normal node.
In a fourth aspect, the present application provides a stream data management apparatus, comprising:
at least one processor, and a memory coupled to the at least one processor;
wherein the memory stores instructions executable by the at least one processor, the at least one processor performing the method of any of the first aspects by executing the instructions stored by the memory.
In a fifth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any of the first aspects.
In a sixth aspect, the present embodiments provide a computer program product comprising instructions which, when run on a computer, cause the method of the first aspect and any of the possible embodiments described above to be implemented.
The technical scheme in the embodiment of the application has the following beneficial effects: according to the database types of the master node and the slave node, adjusting a database protocol supported by the stream data management device; receiving and storing stream data from a master node by using a database protocol of the master node, and then transmitting the stream data to a slave node through the database protocol of the slave node according to a distribution object of the stream data; therefore, data is forwarded between the master node and the slave node through the stream data management device, a strong coupling relation between the master node and the slave node is isolated, and when a plurality of slave nodes exist in the cluster, the master node only needs to send one data, so that the data sending pressure at the master node is reduced.
Drawings
Fig. 1 is an application scenario schematic diagram of a stream data management method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a stream data management apparatus according to an embodiment of the present application;
fig. 3 is a flow chart of a method for managing stream data according to an embodiment of the present application;
Fig. 4 is a flow chart of a method for managing stream data according to an embodiment of the present application;
fig. 5 is a schematic diagram of communication interaction between a database cluster and a stream data management device according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a stream data management apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a stream data management apparatus according to an embodiment of the present application.
Detailed Description
Technical terms related to the embodiments of the present application are described below.
1. Streaming data refers to a set of sequential, large, fast, consecutively arriving data sequences. Streaming data can also be viewed as a dynamic collection of data that continues to grow indefinitely over time. Streaming data is typically data that is continuously generated by thousands of data sources, and is also typically sent in the form of data records at the same time, on a small scale (on the order of a few kilobytes). The streaming data includes a variety of data, such as mobile or web application generated log files used by the customer, online shopping data, in-game player activity, social networking site information, financial transaction lobby or geospatial services, and telemetry data from connected devices or instruments within the data center.
For the purposes, technical solutions and advantages of the present application to become more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.
In the prior art, relational data typically selects either the Postgresql database schema or the Mysql database. The master-slave architecture scheme of the Postgresql database can realize a master-slave architecture based on wal stream replication technology, relevant information of clusters of the Postgresql master-slave architecture is stored in one DCS cluster in a concentrated mode, and then the consistency of parameters and state information of the postgreSQL cluster is ensured by using a distributed consistency algorithm of the DCS cluster; in addition, the configuration parameters of the PostgreSQL database stored in the DCS cluster can be dynamically acquired, set, and modified through REST API commands. The master-slave architecture scheme of the Mysql database can realize a master-slave architecture based on the binary binlog of the Mysql database; and the objective of automatically deploying mysql clusters is achieved by utilizing an active script, which comprises the actions of system environment inspection, related catalog initialization, necessary parameter setting and the like.
But a strong coupling relationship exists between master nodes and slave nodes in the postgresql cluster and the Mysql cluster; if multiple slave nodes need to be deployed, the master node needs to synchronize data to multiple slave nodes at the same time, which increases the pressure of the master node and increases the delay of the transaction on the master node.
Therefore, the application provides a stream data management method, a database cluster, a device and equipment, which are used for solving the technical problem that the strong coupling relation between the master node and the slave node cannot be isolated in the prior art.
Fig. 1 is a schematic application scenario diagram of a stream data management method according to an embodiment of the present application. As shown in fig. 1, the schematic diagram comprises a streaming data management device 120, a first database cluster 110 and a second database cluster 130. Wherein the first database cluster 110 comprises a first database 111, a second database 112 and a third database 113, and the second database cluster 130 comprises a fourth database 131 and a fifth database 132. Wherein the stream data management apparatus 120 may be in wired or wireless communication with each of the first database 111, the second database 112, the third database 113, the fourth database 131, and the fifth database 132.
The streaming data management device 120 may be a device having data processing and storage functions, such as a server cluster, in particular, such as a consul cluster. For example, the stream data management device 120 may use a consul cluster composed of three nodes to store the managed cluster related information, and use the distributed shift protocol used by the consul cluster to ensure that the cluster information data is highly available.
The database types of any one of the first database 111, the second database 112, the third database 113, the fourth database 131, and the fifth database 132 may be relational databases, such as postgreql open source database and mysql open source database.
Wherein, in the first database cluster 110, the first database 111 may be regarded as a master database (may be simply referred to as a master database) in the first database cluster 110, and the second database 112 and the third database 113 may be regarded as slave databases (may be simply referred to as slave databases) in the first database cluster 110. In the second database cluster 130, the fourth database 131 may be considered a master database in the second database cluster 130 and the fifth database 132 may be considered a slave database in the second database cluster 130. The server corresponding to the master library may also be regarded as a master node in the database cluster, and the server corresponding to the slave library may also be regarded as a slave node in the database cluster.
The functions of the streaming data management device 120 will be exemplified below in connection with a schematic structural diagram of the streaming data management device 120 shown in fig. 2.
As shown in fig. 2, the stream data management apparatus 120 includes a stream data transceiving unit 210, a database cluster information management unit 220, and a database cluster function management unit 230. The stream data transceiver unit 210 is configured to receive or transmit stream data; the database cluster information management unit 220 is configured to store database cluster information; the database cluster function management unit 230 is configured to determine whether the database cluster is operating normally.
In a possible implementation manner, the stream data management device 120 may further include a display unit 240, where the display unit 240 is configured to provide an operation and maintenance operation interface to the outside.
Fig. 3 is a schematic flow chart of a method for managing stream data according to an embodiment of the present application.
The stream data management method is applied to the stream data management device in the relational database cluster, the stream data management device is connected to the master node and the slave node in the relational database cluster at the same time, and the master node and the slave node exchange data with the stream data management device. The stream data management apparatus shown in fig. 3 is, for example, the stream data management apparatus 120 shown in fig. 1. The master node shown in fig. 3 is, for example, a server corresponding to the first database 111 or the fourth database 131 shown in fig. 1. The slave node in fig. 3 is, for example, a server corresponding to at least one of the second database 112, the third database 113, and the fifth database 132 shown in fig. 1. For example, if the master node is a server corresponding to the first database 111, the slave node is a server corresponding to the second database 112 and/or the third database 113 shown in fig. 1. If the master node is the server corresponding to the fourth database 131, the slave node is the server corresponding to the fifth database 132 shown in fig. 1.
S301, the stream data management device adjusts a database protocol supported by the stream data management device according to the database types of the master node and the slave node;
the master node is used for receiving the data and storing the data into the master node. The data in the master node can be received from other devices or input by a user, and the data in the master node can provide read-write service to the outside. The slave node is used for backing up the data stored by the master node, in other words, the slave node is actually a backup database of the master node, and the data in the slave node can only provide read-only service to the outside. The number of slave nodes may be one or more, which is not limited in the embodiment of the present application.
The specific meaning of the database type may be referred to in the foregoing description, and will not be described here.
Specifically, the stream data management device determines the database types corresponding to the master node and the slave node, determines a database protocol corresponding to the database type according to the database types, and configures the database protocol. A database protocol may also be understood as a reception protocol for receiving streaming data.
For example, if the stream data management apparatus determines that the database type is a postgresql open source database and that the database protocol corresponding to the postgresql open source database is the stream replication protocol corresponding to postgresql, the stream replication protocol corresponding to postgresql is configured in the stream data management apparatus. If the stream data management device determines that the database type is a mysql open source database and determines that the database protocol corresponding to the mysql open source database is a mysql-binlog protocol, configuring the mysql-binlog protocol in the stream data management device.
S302, the stream data management device receives and stores stream data from the master node by using the database protocol of the master node, and then sends the stream data to the slave node through the database protocol of the slave node according to the distribution object of the stream data.
The stream data may be a log file, a control file, a parameter file, or a Write Ahead Log (WAL) file of the master node.
The master node transmits stream data to the stream data management apparatus, and the stream data management apparatus determines a database protocol of the master node and receives stream data from the master node based on the database protocol of the master node.
Specifically, when the master node transmits stream data to the slave node, the stream data management device uses itself as a virtual slave node corresponding to the master node, and receives the stream data through a database protocol.
The stream data management device determines a distribution object of stream data and sends the stream data to the distribution object, wherein the distribution object of the stream data is a slave node binding a master-slave relationship with the master node.
For example, the stream data management apparatus prestores information of a slave node binding a master-slave relationship with the master node, or receives information of a slave node binding a master-slave relationship with the master node from the master node. Further, the stream data management apparatus may determine a distribution object of the stream data from information of the slave node binding the master node in a master-slave relationship. Hereinafter, for convenience of description, a distribution object of stream data is referred to as a slave node.
The stream data management device determines the database type and the corresponding database protocol of the slave node, and transmits stream data to the slave node based on the database protocol of the slave node.
In this case, the stream data management apparatus is one slave node of the master node for the master node. For a slave node, the stream data management apparatus is the master node of the slave node.
The stream data management apparatus may further transmit stream data to the slave node according to a kind of a configured mode, wherein the kind of the mode includes a synchronous mode and an asynchronous mode. The synchronization mode refers to transmitting stream data to one slave node at the stream data management device and feeding back a response message indicating that the one slave node has successfully received stream data to the master node, transmitting stream data to the next slave node. The asynchronous mode refers to that after the stream data management device sends stream data to one slave node, the stream data is continuously sent to the next slave node, and the response message is fed back to the master node.
In another possible embodiment, the stream data management device determines that there is no slave node binding a master-slave relationship with the master node, in which case the stream data management device determines itself as a virtual node, which may be regarded as a run-free node, and automatically returns a message to the master node that the stream data was successfully received when the virtual node is scheduled. After receiving the stream data from the master node, the stream data management apparatus stores the stream data in the stream data management apparatus and transmits a response message to the master node, the response message indicating that the slave node has successfully received the stream data. And when the newly added slave node and the master node are bound in a master-slave relationship, the stream data management device sends the stored stream data to the newly added slave node.
Alternatively, the flow data management device determines that the slave node is in a maintenance state, and may also be understood as a slave node shutdown to perform maintenance tasks, such as a system kernel upgrade. In this case, the stream data management apparatus sets the slave node as a virtual node. When the stream data management apparatus transmits stream data to the slave node, and determines that the slave node is a virtual node, the stream data is temporarily stored in the stream data management apparatus, and a response message indicating that the slave node has successfully received the stream data is transmitted to the master node.
After the stream data management apparatus determines that the slave node has operated normally, the slave node is modified to a normal node and stream data temporarily stored in the stream data management apparatus is transmitted to the slave node.
In this embodiment, by setting the virtual node, even in the case where the slave node cannot normally receive the stream data from the master node, the master node can normally receive the response message that the slave node has successfully received the stream data. In other words, the master node cannot perceive the change of the slave node, i.e., the association relationship between the master node and the slave node is decoupled.
Before executing S302, the stream data management apparatus also needs to determine whether the database cluster where the master node is located is operating normally. The manner in which the streaming data management device determines whether the database cluster is operating properly will be illustrated in connection with a flow chart of a streaming data management method shown in fig. 4.
S401, the stream data management device acquires pre-stored database cluster information.
The database cluster information may be stored in a database cluster information management table of the stream data management apparatus. The database cluster information includes the name of the database cluster, the database type, the internet protocol address (internet protocol address, IP) address of each database (node) in the database cluster, the virtual IP address, the data storage directory, the port, the database status (node status), the stream data reception location, the role, and the like. Wherein the virtual IP address is used to bind a master node in a database cluster. One or more virtual IP addresses may be included in the stream data management device, one virtual IP address corresponding to each database cluster. In other words, the stream data management device may store database cluster information of one or more database clusters. The database cluster information stored in the streaming data management apparatus is exemplified by a cluster information management table shown in table 1.
TABLE 1
The database type of the database cluster B is mysql, the database cluster B comprises a master node and a slave node, the IP address of the master node in the database cluster B is 10.11.1.1, the virtual IP address is 10.11.1.9, the data directory of stream data is/data/mysql, the port is 3306, the state is normal, and the stream data receiving position of the master node of the database cluster B is 0/200A93C8. The IP address of the slave node in the database cluster B is 10.11.1.2, the data catalog of the stream data is/data/mysql, the port is 3306, and the state is normal.
S402, the stream data management apparatus determines whether a data directory and a port exist.
In particular, the stream data management apparatus may check the data directory and the ports in the database cluster by executing the command or using the tool, and confirm whether the data directory and the ports exist. The commands are, for example, structured query language (structured query language, SQL) statements, and the tools are, for example, SQL server (server) configuration tools.
If the stream data management apparatus determines that the data directory and port in the database cluster information do not exist, S403 is performed, i.e., the stream data management apparatus initializes new database cluster information. If the stream data management apparatus determines that the data object and port in the database cluster information exist, S404 is performed, i.e., the stream data management apparatus checks whether the database cluster status is normal.
When the stream data management apparatus executes S403, the stream data management apparatus obtains new database cluster information by re-acquiring the information of the master node and the slave node from the database cluster through the IP addresses and the administrator account numbers of the master node and the slave node in the pre-stored database cluster. And the stream data management device updates the database cluster information table according to the new database cluster information to obtain an updated database cluster information table. For example, the stream data management apparatus transmits a request message for requesting status information, port information, data directory information, etc. of a master node to the master node according to a pre-stored IP address of the master node. And the stream data unit sends a request message to each node based on the pre-stored IP address of each node in the database cluster, so that the database cluster information of the database cluster can be obtained.
It should be noted that, the steps S401 to S403 may be specifically implemented by a database cluster information management unit in the stream data management apparatus, for example, the database cluster information management unit 220 shown in fig. 2.
When the stream data management apparatus performs S404, the stream data management apparatus may detect the database cluster state through the heartbeat packet. Specifically, the stream data management device may send heartbeat packets to the master node and the slave node in the database cluster for status monitoring. The master node and the slave node receive heartbeat packets from the stream data management device, generate state information based on the current node state and return the state information to the stream data management device, wherein the node state comprises normal and abnormal states. The stream data management device determines the state of each node in the database cluster according to the state information returned by each node in the database cluster, and synchronizes the received state information into the database cluster information management table.
If one of the plurality of state information received by the stream data management apparatus indicates that the state of the node is abnormal, it is determined that the state of the database cluster is abnormal. If the node state indicated by each of the plurality of state information received by the stream data management device is normal, determining that the state of the database cluster is normal.
If the stream data management apparatus determines that the state of the database cluster is normal, S405 is performed, i.e., the stream data management apparatus determines that the database cluster has successfully bound the virtual IP address. If the stream data management apparatus determines that the state of the database cluster is abnormal, S406 is performed, i.e., the stream data management apparatus determines whether a failure occurs in a master node in the database cluster.
At the execution of S405, the stream data management device determines that the database cluster has successfully bound the virtual IP address, specifically, may determine that the master node in the database cluster has successfully bound the virtual IP address.
In performing S406, the stream data management apparatus may determine whether the master node in the database cluster fails from the status information fed back by the master node. For example, if the streaming data management device parses the state information from the master node, determines that the current state of the master node is normal, then it is determined that the master node has not failed. If the streaming data management device determines that the current state of the master node is abnormal (abnormal), it is determined that the master node has failed.
If the stream data management apparatus determines that the master node in the database cluster has not failed, S407 is performed, i.e. the stream data management apparatus determines that the slave node in the database cluster has failed. If the stream data management apparatus determines that the master node in the database cluster has failed, S408 is performed, i.e. the stream data management apparatus marks the failed master node as a failed node.
After S407 is performed, S409 is performed, and the stream data management apparatus marks the failed slave node as a failed node.
Specifically, the stream data management apparatus marks the information of the failed slave node in the database cluster information. For example, the failed slave node is marked as failed or failed node at its name.
S410, the stream data management device sends the data on the main node in the database cluster to the fault node so as to enable the fault node to be recovered to be normal.
The stream data management device acquires the data on the main node and sends the data to the fault node, and the fault node can recover the data after receiving the data from the stream data management device, thereby recovering the data to be normal. The data acquired by the stream data management device from the master node may be all data in the master node, or may be part of the data acquired by the stream data management device from the master node, for example, the stream data management device acquires only the data updated after the last update point, where the update point is the point in time when the stream data management device acquires the data from the master node.
After executing S408, S411 is executed, and the stream data management apparatus shields the failed node from the database cluster.
And the stream data management device determines that the fault node is a master node, unbinding the fault node and the virtual IP address, removing the database cluster, and shielding the fault node from offline.
In one possible implementation, the flow data management device may re-register the repaired master node into the database cluster after the failed node is repaired and brought online. The re-registered repaired master node will operate as a slave node in the database cluster.
S412, the stream data management device determines a slave node with normal state from the database cluster as a new master node.
The stream data management apparatus selects one slave node from among the slave nodes in the database cluster as a new master node, and binds the virtual IP address with the new master node.
It should be noted that, the steps S404 to S412 may be specifically performed by a database cluster management unit in the stream data management apparatus, and the database cluster management unit is, for example, the database cluster management unit 230 shown in fig. 2.
S413, the stream data management apparatus updates the database cluster information.
After determining that the database cluster has a fault node, the stream data management device updates the database cluster information and determines whether a master-slave node is switched or a node is off line. The manner in which the stream data management device updates the database cluster information may correspond to the content described with reference to S403, which is not described herein.
It should be noted that S413 may be specifically implemented by a database cluster information module in the stream data management apparatus, for example, the database cluster information unit 220 shown in fig. 2.
After executing S413, then executing S405, the stream data management apparatus determines that the database cluster has successfully bound the virtual IP address.
The steps S401-S413 are the contents of checking and updating the stored database cluster information at preset intervals by the stream data management device, and if the stream data management device determines that the database cluster information is abnormal when executing S401, that is, executing S414, that is, the stream data management device determines that the database cluster information is abnormal, then S402-S413 need not be executed.
In the stream data management methods shown in fig. 3 and 4, an example of one database cluster is described. The flow data management method provided in the embodiment of the present application will be described below with reference to a schematic communication interaction between a database cluster and a flow data management device shown in fig. 5, where the database cluster includes a first database cluster and a second database cluster.
As shown in fig. 5, the schematic diagram includes a first database cluster, a second database cluster, and a stream data management device. The first database cluster comprises a first master node, a first slave node and a second slave node. The second database cluster includes a second master node and a third slave node.
The first database cluster shown in fig. 5 is, for example, the first database cluster 110 shown in fig. 1, the first master node is, for example, a server corresponding to the first database 111 shown in fig. 1, the first slave node is, for example, a server corresponding to the second database 112 shown in fig. 1, and the second slave node is, for example, a server corresponding to the third database 113 shown in fig. 1. The second database cluster shown in fig. 5 is, for example, the second database cluster 130 shown in fig. 1, the second master node is, for example, a server corresponding to the fourth database 131 shown in fig. 1, and the third slave node is, for example, a server corresponding to the fifth database 132 shown in fig. 1. The stream data management apparatus shown in fig. 4 is, for example, the stream data management apparatus 120 shown in fig. 1.
S501, the first master node transmits first stream data to the stream data management apparatus. Accordingly, the stream data management apparatus receives the first stream data from the first master node.
S502, the stream data management apparatus transmits the first stream data to the first slave node. Accordingly, the first slave node receives the first stream data from the stream data management apparatus.
S503, the first slave node transmits a first response message to the stream data management apparatus. Accordingly, the stream data management apparatus receives a first response message from the first slave node.
Wherein the first response message indicates that the first slave node has successfully received the first stream data.
S504, the stream data management apparatus transmits a first response message to the first master node. Accordingly, the first master node receives the first response message from the stream data management device and determines that the first slave node has successfully received the first stream data.
S505, the stream data management apparatus transmits the first stream data to the second slave node. Accordingly, the second slave node receives the first stream data from the stream data management apparatus.
S506, the second slave node transmits a second response message to the stream data management apparatus. Accordingly, the stream data management apparatus receives a second response message from the second slave node.
Wherein the second response message indicates that the second slave node has successfully received the first stream data.
S507, the stream data management apparatus transmits a second response message to the first master node. Accordingly, the first master node receives the second response message from the stream data management device and determines that the second slave node has successfully received the first stream data.
S508, the second master node transmits the second stream data to the stream data management apparatus. Accordingly, the stream data management device receives the second stream data from the second master node.
It should be noted that, steps S501 to S507 are the contents of the stream data management device managing the stream data of the first database cluster, and steps S502 to S505 may be sequentially executed, or after S502 is executed, S505 is executed first, and S503 and S504 are executed, specifically, the modes related to the modes configured by the stream data management device itself, for example, a synchronous mode and an asynchronous mode. The specific meaning of the synchronous mode and the asynchronous mode may be referred to as described above at S302.
S509, the stream data management apparatus transmits the second stream data to the third slave node. Accordingly, the third slave node receives the second stream data from the stream data management apparatus.
S510, the third slave node transmits a third response message to the stream data management apparatus. Accordingly, the stream data management apparatus receives a third response message from a third slave node.
Wherein the third response message indicates that the third slave node has successfully received the second stream data.
S511, the stream data management apparatus transmits a third response message to the second master node. Accordingly, the second master node receives a third response message from the stream data management device, and determines that the third slave node has successfully received the second stream data.
It should be noted that, the steps S508-S511 described above are the content of the second stream data of the second database cluster managed by the stream data management device, where the steps S508-S511 may be performed synchronously with the steps S501-S507 described above, or performed before the steps S501-S507, or performed after the steps S501-S507, which is not limited in the embodiments of the present application.
Based on the same inventive concept, the present application also provides a stream data management device, which connects a master node and a slave node in a relational database cluster, and the master node and the slave node exchange data through the stream data management device. The stream data management apparatus is used to implement any of the above-described stream data management methods, for example, the stream data management method shown in fig. 3, and may also implement the functions of the foregoing stream data management apparatus.
Referring to fig. 6, a schematic structural diagram of a stream data management apparatus according to an embodiment of the present application is provided, and as shown in fig. 6, the stream data management apparatus 600 includes a protocol synchronization module 601 and a data distribution module 602.
Illustratively, the protocol synchronization module 601 is configured to adjust a database protocol supported by the stream data management device according to a database type to which the master node and the slave node belong;
The data distribution module 602 is configured to receive and store stream data from a master node using a database protocol of the master node, and then send the stream data to a slave node through the database protocol of the slave node according to a distribution object of the stream data.
In one possible implementation, the protocol synchronization module 601 is configured to:
establishing a virtual slave node to connect with the master node, acquiring a database protocol to which the master node belongs, and adding the database protocol to a database protocol supported by the stream data management device;
and connecting the slave node as a master node, acquiring a database protocol to which the slave node belongs, and adding the database protocol to the database protocol supported by the stream data management device.
In one possible implementation, the protocol synchronization module 601 is configured to:
sending heartbeat packets to all the master nodes and slave nodes by using the adjusted database protocol, receiving the state information returned by each node and updating the state information into a database cluster information management table;
judging whether a fault node exists according to the state information, and if so, marking the fault node in a database cluster information management table.
In one possible implementation, the protocol synchronization module 601 is configured to:
If the marked fault node is the master node, shielding the fault node out of the database cluster, and re-registering the fault node into the database cluster after the fault node is repaired and online;
if the marked fault node is the slave node, acquiring data in the slave node from the master node, and synchronizing the data in the slave node to the fault node so as to restore the fault node to be a normal node.
It should be noted that, the protocol synchronization module 601 in the stream data management apparatus 600 may also implement the functions of the database cluster information management unit 220 and the database cluster function management unit 230 shown in fig. 2, and the data distribution module 602 may also implement the functions of the stream data transceiver unit 210 shown in fig. 2.
Based on the same inventive concept, the embodiments of the present application provide a stream data management apparatus for implementing any of the above-described stream data management methods, for example, the stream data management method shown in fig. 3, and may also implement the functions of the foregoing stream data management device.
Referring to fig. 7, a schematic structural diagram of a stream data management apparatus according to an embodiment of the present application is provided, and as shown in fig. 7, the stream data management apparatus 700 includes at least one processor 701, where the processor 701 is configured to implement the steps of the stream data management method according to the embodiment of the present application when executing a computer program stored in a memory 702.
Alternatively, the processor 701 may be a central processing unit, an application specific integrated circuit (english: application Specific Integrated Circuit, abbreviated as ASIC), or one or more integrated circuits for controlling program execution.
Optionally, the device for protecting data integrity further includes a Memory 702 connected to the at least one processor 701, where the Memory 702 may include a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM) and a disk Memory. The memory 702 is used for storing data required by the processor 701 when running, i.e. instructions executable by the at least one processor 701 are stored, and the at least one processor 701 performs the method as shown in fig. 3 above by executing the instructions stored by the memory 702. Wherein the number of memories 702 is one or more.
The embodiments also provide a computer storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the steps of the method of stream data management as above.
The embodiments of the present application provide a computer program product comprising instructions which, when run on a computer, cause a stream data management method as any one of the above, for example the stream data management method shown in fig. 3, to be implemented.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.
Claims (10)
1. A stream data management method applied to a stream data management device in a relational database cluster, the stream data management device connecting a master node and a slave node in the relational database cluster, the master node and the slave node exchanging data through the stream data management device, the method comprising:
according to the database types of the master node and the slave node, adjusting a database protocol supported by the stream data management device;
and receiving and storing stream data from the master node by using the database protocol of the master node, and then sending the stream data to the slave node through the database protocol of the slave node according to the distribution object of the stream data.
2. The method of claim 1, wherein adjusting the database protocol supported by the stream data management apparatus according to the database type to which the master node and the slave node belong comprises:
establishing a virtual slave node to connect with the master node, acquiring a database protocol to which the master node belongs, and adding the database protocol to a database protocol supported by the stream data management device;
and connecting the slave node as a master node, acquiring a database protocol to which the slave node belongs, and adding the database protocol to the database protocol supported by the stream data management device.
3. The method of claim 1, wherein after adjusting the database protocol supported by the stream data management apparatus, comprising:
sending heartbeat packets to all the master nodes and slave nodes by using the adjusted database protocol, receiving the state information returned by each node and updating the state information into a database cluster information management table;
judging whether a fault node exists according to the state information, and if so, marking the fault node in a database cluster information management table.
4. The method of claim 3, wherein after marking the failed node in the database cluster information management table, comprising:
if the marked fault node is a master node, shielding the fault node out of the database cluster, and re-registering the fault node into the database cluster after the fault node is repaired and online;
and if the marked fault node is a slave node, acquiring data in the slave node from a master node, and synchronizing the data into the fault node so as to restore the fault node to be a normal node.
5. A database cluster, comprising:
the master node is used for issuing data and configuring slave nodes in the database cluster;
A slave node for receiving and storing data;
stream data management means connecting all of said master nodes and said slave nodes and performing the method according to any of claims 1-4.
6. The database cluster of claim 5, wherein the stream data management means is for:
when the slave node enters a maintenance state, temporarily storing the data sent to the slave node into the stream data management device, and sending a response message to the master node, wherein the response message indicates that the slave node has successfully received the data;
and sending the data temporarily stored in the stream data management device to the slave node until the maintenance of the slave node is finished.
7. The database cluster of claim 5, wherein the stream data management means is for:
storing a database cluster information management table, and setting an IP address for the master node according to the database cluster information management table; the database cluster information management table comprises state information of all nodes.
8. A stream data management apparatus, comprising:
the protocol synchronization module is used for adjusting the database protocol supported by the stream data management device according to the database types of the master node and the slave node;
And the data distribution module is used for receiving and storing the stream data from the master node by using the database protocol of the master node, and then sending the stream data to the slave node through the database protocol of the slave node according to the distribution object of the stream data.
9. A stream data management apparatus, characterized by comprising:
at least one processor, and
a memory coupled to the at least one processor;
wherein the memory stores instructions executable by the at least one processor, the at least one processor performing the method of any of claims 1-4 by executing the instructions stored by the memory.
10. A readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the stream data management method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310229323.7A CN116360920A (en) | 2023-03-07 | 2023-03-07 | Stream data management method, database cluster, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310229323.7A CN116360920A (en) | 2023-03-07 | 2023-03-07 | Stream data management method, database cluster, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116360920A true CN116360920A (en) | 2023-06-30 |
Family
ID=86912132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310229323.7A Pending CN116360920A (en) | 2023-03-07 | 2023-03-07 | Stream data management method, database cluster, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116360920A (en) |
-
2023
- 2023-03-07 CN CN202310229323.7A patent/CN116360920A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6684367B2 (en) | Data processing method and device | |
| CN102012944B (en) | Distributed NOSQL (not only structured query language) database capable of providing replication property | |
| EP1873645A1 (en) | Storage system and data replication method | |
| US11640261B2 (en) | Log processing method to avoid log collision, and related device and system | |
| CN111404730B (en) | State synchronization method and device of virtual router, electronic equipment and storage medium | |
| WO2016177231A1 (en) | Dual-control-based active-backup switching method and device | |
| CN102402441A (en) | System and method for configuring multiple computers | |
| US20210089379A1 (en) | Computer system | |
| JP4320314B2 (en) | Computer system, synchronization processing method, and program | |
| KR101527634B1 (en) | Method and apparatus for providing sharding service | |
| CN106855869B (en) | Method, device and system for realizing high availability of database | |
| CN114297296A (en) | Server active-standby switching system for broadcasting | |
| JP4550604B2 (en) | Setting information synchronization program | |
| CN113190620A (en) | Method, device, equipment and storage medium for synchronizing data between Redis clusters | |
| CN105323271B (en) | Cloud computing system and processing method and device thereof | |
| CN116360920A (en) | Stream data management method, database cluster, device and equipment | |
| CN115794769B (en) | Method for managing high-availability database, electronic equipment and storage medium | |
| WO2011099380A1 (en) | Essential data management system, calculator, essential data management program, recording medium, and communication method | |
| CN105511808A (en) | Data operation method, system and related device | |
| JP2012133417A (en) | Information processing system | |
| US20220342775A1 (en) | Storage system, storage node virtual machine restore method, and recording medium | |
| WO2022222403A1 (en) | Task distribution system, method, and apparatus, computer device, and storage medium | |
| JP2011002970A (en) | Distributed data management system, data management device, data management method, and program | |
| JP5956940B2 (en) | Redundant system and working machine determination method | |
| CN113890875B (en) | Task allocation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |