CN112612853A - Data processing method and device based on database cluster and electronic equipment - Google Patents
Data processing method and device based on database cluster and electronic equipment Download PDFInfo
- Publication number
- CN112612853A CN112612853A CN202011588369.0A CN202011588369A CN112612853A CN 112612853 A CN112612853 A CN 112612853A CN 202011588369 A CN202011588369 A CN 202011588369A CN 112612853 A CN112612853 A CN 112612853A
- Authority
- CN
- China
- Prior art keywords
- database
- operation log
- log
- slave
- slave database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0778—Dumping, i.e. gathering error/state information after a fault for later diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application is applicable to the technical field of cloud storage, and provides a data processing method and device based on a database cluster and an electronic device, wherein the method comprises the following steps: acquiring an operation log of a master database in a Redis database cluster, and sequentially distributing the operation log to each slave database in the Redis database cluster; when a first operation log which is distributed by a first slave database at the latest is not continuous with a latest operation log stored locally, acquiring a first operation log synchronization request sent by the first slave database; according to the first operation log synchronization request, acquiring a first target operation log with a distribution sequence between the first operation log and the latest operation log; distributing the first target oplog to the first slave database. The scheme can ensure the consistency of the data in the slave database and the data in the master database.
Description
Technical Field
The application belongs to the technical field of cloud storage, and particularly relates to a data processing method and device based on a database cluster and an electronic device.
Background
A Remote Dictionary service (Redis) database cluster has gradually replaced the original cache system due to the fast read-write speed and stable performance, and is widely used by various systems.
The Redis database cluster is divided into a master database and a slave database, and data in the slave database needs to be consistent with data in the master database so as to ensure normal operation of functions such as reading and writing in the Redis database cluster.
At present, abnormal conditions, such as shutdown of a main database in Redis or interruption of communication between a main database and a slave database, can cause the problems of inconsistent data of the main database and the slave database or unavailability of the Redis database for a long time.
The current conventional solutions: 1. manual intervention is needed for repairing and restarting database faults, but the recovery service is long in time consumption; 2. a slave database capable of normal communication is used as a master database, but time difference exists in the synchronization process of the master database and the slave database, so that data are inconsistent, and normal business requirements cannot be met.
Disclosure of Invention
The embodiment of the application provides a data processing method and device based on a database cluster and electronic equipment, and aims to solve the problems that in the prior art, the service recovery time is long when a Redis main database is down, and data are inconsistent due to time difference in the synchronization process of the main database and the slave database.
A first aspect of an embodiment of the present application provides a data processing method based on a database cluster, including:
acquiring an operation log of a master database in a Redis database cluster, and sequentially distributing the operation log to each slave database in the Redis database cluster;
when a first operation log which is distributed by a first slave database latest is not continuous with a second operation log which is distributed last, a first operation log synchronization request sent by the first slave database is obtained;
according to the first operation log synchronization request, acquiring a first target operation log with a distribution sequence between the first operation log and the second operation log;
distributing the first target oplog to the first slave database.
A second aspect of an embodiment of the present application provides a database cluster-based data processing apparatus, including:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring an operation log of a master database in a Redis database cluster and sequentially distributing the operation log to each slave database in the Redis database cluster;
the second acquisition module is used for acquiring a first operation log synchronization request sent by a first slave database when a first operation log which is distributed latest by the first slave database is discontinuous with a second operation log which is distributed last time;
a third obtaining module, configured to obtain, according to the first operation log synchronization request, a first target operation log whose distribution order is located between the first operation log and the second operation log;
a distribution module, configured to distribute the first target operation log to the first slave database.
A third aspect of embodiments of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to the first aspect when executing the computer program.
A fourth aspect of embodiments of the present application provides a computer-readable storage medium, in which a computer program is stored, which, when executed by a processor, performs the steps of the method according to the first aspect.
A fifth aspect of the present application provides a computer program product, which, when run on an electronic device, causes the electronic device to perform the steps of the method of the first aspect described above.
As can be seen from the above, in the embodiment of the present application, operation logs of a master database in a Redis database cluster are obtained, the operation logs are sequentially distributed to each slave database in the Redis database cluster, when a first operation log that is newly distributed to a first slave database is not consecutive to a second operation log that is distributed last, a first operation log synchronization request sent by the first slave database is obtained, according to the first operation log synchronization request, a first target operation log whose distribution order is located between the first operation log and the second operation log is obtained, and the first target operation log is distributed to the first slave database. The process realizes real-time monitoring on the master database by introducing a real-time monitoring component, and when the operation log which is distributed latest in the slave database is not continuous with the operation log which is stored latest locally, the missing operation log is found out according to the distribution sequence of the operation logs and is distributed to the slave database, so that the consistency of the data in the slave database and the data in the master database is ensured.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a first flowchart of a data processing method based on a database cluster according to an embodiment of the present application;
fig. 2 is a flowchart ii of a data processing method based on a database cluster according to an embodiment of the present application;
fig. 3 is a block diagram of a data processing apparatus based on a database cluster according to an embodiment of the present application;
fig. 4 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
In particular implementations, the electronic devices described in embodiments of the present application include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers having touch sensitive surfaces (e.g., touch screen displays and/or touch pads). It should also be understood that in some embodiments, the device is not a portable communication device, but is a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or touchpad).
In the discussion that follows, an electronic device that includes a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may include one or more other physical user interface devices such as a physical keyboard, mouse, and/or joystick.
The electronic device supports various applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disc burning application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, an exercise support application, a photo management application, a digital camera application, a web browsing application, a digital music player application, and/or a digital video player application.
Various applications that may be executed on the electronic device may use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the electronic device can be adjusted and/or changed between applications and/or within respective applications. In this way, a common physical architecture (e.g., touch-sensitive surface) of the electronic device may support various applications with user interfaces that are intuitive and transparent to the user.
It should be understood that, the sequence numbers of the steps in this embodiment do not mean the execution sequence, and the execution sequence of each process should be determined by the function and the inherent logic of the process, and should not constitute any limitation to the implementation process of the embodiment of the present application.
In order to explain the technical solution described in the present application, the following description will be given by way of specific examples.
Referring to fig. 1, fig. 1 is a first flowchart of a data processing method based on a database cluster according to an embodiment of the present application. The data processing method based on the database cluster is specifically realized by an implementation monitoring component, or is realized by other designed hardware or software.
As shown in fig. 1, a data processing method based on database cluster includes the following steps:
In the Redis database cluster structure, databases are divided into two categories: a master database (master) and a slave database (slave). One master may have multiple slaves, but one slave can only correspond to one master.
In this step, the data in the slave database is synchronized, specifically, by monitoring the operation log acquired by the component from the master database in real time. The operation logs are transmitted between the master database and the slave database through the real-time monitoring assembly, so that the slave database can synchronously update data of the slave database based on the operation logs of the master database.
The operation log records data updating operation of the master database, and when the slave data acquires the distributed operation log, local data updating can be performed based on the data updating operation in the operation log, wherein the data updating operation includes but is not limited to data deletion, data new creation, data modification, establishment of relationships between data, data writing operation, creation of a data table, import of the data table and the like.
The real-time monitoring component may store the operation logs locally after acquiring the operation logs, specifically, delete the operation logs stored locally after keeping the operation logs for a set time, or maintain the number of the operation logs kept at a set value, and delete the operation logs acquired first when the number of the operation logs exceeds the set value.
Specifically, the operation log of the master database in the Redis database cluster is obtained according to a preset frequency, or obtained when a new operation log is detected to be generated in the master database, or obtained once when a data updating operation is detected to occur in the master database, whenever the data updating operation is monitored, in the master database.
As an optional implementation manner, the acquiring an operation log of a master database in a Redis database cluster, and sequentially distributing the operation log to each slave database in the Redis database cluster includes:
acquiring an operation log of a master database in a Redis database cluster; carrying out serial number marking on the operation logs according to the acquisition sequence to obtain log numbers; and sequentially distributing the operation logs containing the log numbers to each slave database in the Redis database cluster.
The log numbers are numbers which are sequenced according to the sequence of obtaining the operation logs. The log numbers correspond to the acquisition sequence of the operation logs one by one.
When the real-time monitoring component acquires the operation logs of the master database, the sequence numbers of the acquired operation logs need to be marked, the marked sequence numbers are used for marking the acquisition sequence of the operation logs, and the operation logs are distributed to the slave database in sequence according to the acquisition sequence of the operation logs.
Specifically, the master database is required to start a log function, and each operation log of the master database acquired by the real-time monitoring component generates a unique serial number topicid with a sequence. The log function needs to be started from the database as well. And the real-time monitoring component is configured with the ip of the master library and the ip of the slave library, so that information transmission between the master library and the slave library is realized, and logs of the master database are extracted in real time and distributed to the slave database.
When the first slave database receives the first operation log which is currently and newly distributed, the first slave database does not store the first operation log locally, and needs to judge the relationship between the first operation log which is newly distributed and the second operation log which is distributed last, and judge whether the first operation log and the second operation log are continuous operation logs. When the first slave database determines that the first slave database and the master database are continuous logs, the first slave database locally stores the first operation log and executes the first operation log, so that data consistency between the first slave database and the master database is realized.
When the first operation log which is currently distributed by the first slave database and is not continuous with the second operation log which is distributed last is judged, the log missing between the two adjacent operation logs can be determined.
The secondary database can determine whether the operation log is a continuous log by comparing log numbers of the operation log distributed by the real-time monitoring component and determining whether the log numbers are continuous numbers. And judging whether the log number of the second operation log stored locally and the log number of the first operation log are continuous numbers, if so, determining that the two operation logs are continuous logs, and vice versa.
For example: if the topicid of the database is t-001 and the topicid of the first operation log pushed by the real-time monitoring component is t-003, it can be known that t-003 is generated later than t-001, and it can be determined that an operation log with the topicid of t-002 is missing in the middle, and the currently received first operation log and the locally stored latest operation log are not continuous logs.
The reason for generating the discontinuous operation log may be that the data is lost due to network interruption, or after the original master library is repaired and restarted to become the slave database, the itself recorded topocid is not up to date and thus the data is lost.
Specifically, as an optional implementation manner, the first operation log synchronization request carries a first log number of the first operation log and a second log number of the second operation log; wherein the first log number is not consecutive to the second log number.
That is, the sequence number of the oplog may be distributed along with it when the oplog synchronization request is sent from the database to the real-time monitoring component.
Likewise, the log number of the first target operation log located between the first operation log and the second operation log is between the log number of the first operation log and the log number of the second operation log.
That is, the obtaining of the first target operation log having the distribution order between the first operation log and the second operation log according to the first operation log synchronization request includes:
and according to the first operation log synchronization request, acquiring a first target operation log with the log number between the first log number and the second log number.
Under the condition that the operation log is judged to be missing, the real-time monitoring component acquires a first target operation log with a distribution sequence between the first operation log and the second operation log, namely the target operation log with a log number between the log numbers of the first operation log and the second operation log, and distributes the target operation log to the first slave database, and the first slave database analyzes and sequentially executes the log data after receiving the target operation logs.
For example, the topocid of the latest operation log currently and locally stored in the slave database is t-003, the topocid of the operation log issued last in the real-time monitoring component is t-005, the slave database calls a fresh () interface in the real-time monitoring component, the entry parameter is t-003, the real-time monitoring component circularly takes out the t-004 and t-005 logs in the log record table and returns the t-004 and t-005 logs to the slave database which sends an instruction, the slave database analyzes log data in sequence to carry out synchronous operation, and the topocid of the last operation log is recorded, so that the data stored in the slave database is ensured to be consistent with the data in the master database.
The slave database updates the topicid recording the latest operation log after executing log synchronization operation each time, and is used for ensuring that the data of the slave database and the data of the master database are consistent based on the log number.
The missing operation logs are found and distributed to the slave databases, and consistency of data in the slave databases and data in the master database is guaranteed.
In the embodiment of the application, by acquiring the operation logs of a master database in a Redis database cluster, the operation logs are sequentially distributed to each slave database in the Redis database cluster, when a first operation log which is newly distributed by a first slave database is not continuous with a second operation log which is distributed last, a first operation log synchronization request sent by the first slave database is acquired, according to the first operation log synchronization request, a first target operation log with the distribution sequence between the first operation log and the second operation log is acquired, and the first target operation log is distributed to the first slave database. The process realizes real-time monitoring on the master database by introducing a real-time monitoring component, and when the operation log which is distributed latest in the slave database is not continuous with the operation log which is stored latest locally, the missing operation log is found out according to the distribution sequence of the operation logs and is distributed to the slave database, so that the consistency of the data in the slave database and the data in the master database is ensured.
In the embodiment of the application, by acquiring the operation logs of a master database in a redis database cluster, the operation logs are sequentially distributed to each slave database in the redis database cluster, when a first operation log which is newly distributed to a first slave database is not continuous with a latest operation log which is locally stored, a first operation log synchronization request sent by the first slave database is acquired, according to the first operation log synchronization request, a first target operation log with the distribution sequence between the first operation log and the latest operation log is acquired, and the first target operation log is distributed to the first slave database. The process realizes real-time monitoring on the master database by introducing a real-time monitoring component, and when the operation log which is distributed latest in the slave database is not continuous with the operation log which is stored latest locally, the missing operation log is found out according to the distribution sequence of the operation logs and is distributed to the slave database, so that the consistency of the data in the slave database and the data in the master database is ensured.
The embodiment of the application also provides different implementation modes of the data processing method based on the database cluster.
Referring to fig. 2, fig. 2 is a flowchart ii of a data processing method based on a database cluster according to an embodiment of the present application. As shown in fig. 2, a data processing method based on database cluster includes the following steps:
The implementation process of this step is the same as that of step 101 in the foregoing embodiment, and is not described here again.
The implementation process of this step is the same as that of step 102 in the foregoing embodiment, and is not described here again.
The implementation process of this step is the same as the implementation process of step 103 in the foregoing embodiment, and is not described here again.
The implementation process of this step is the same as that of step 104 in the foregoing embodiment, and is not described here again.
And step 205, when the master database is monitored to have a fault, determining a third operation log which is finally distributed to each slave database, and sequentially accessing a fourth operation log which is finally distributed to each slave database.
And step 206, determining a target slave database with the fourth operation log consistent with the third operation log from the slave database according to the third operation log and the fourth operation log.
When the master database is monitored to have a fault, circularly accessing each slave database in the Redis database cluster in sequence, judging whether the latest operation log recorded in each slave database (namely the fourth operation log distributed last) is consistent with the latest operation log in the real-time monitoring component (namely the third operation log distributed to each slave database last) or not, and if so, setting the current slave database as a new master database.
In the process, the real-time monitoring component monitors whether the main database component works normally or not in real time. When the real-time monitoring component monitors the downtime of the main database in real time, all configured databases ip are immediately circulated from the configuration files, whether the topicid of the slave database is consistent with the self topicid is compared, and if the topicid is consistent, the current slave database is set as a new main database.
If not, the next slave library ip is compared circularly. The reason for the inconsistency may be that the original problematic recovery from the library was used, or that the latest oplog was not received from the library due to a network failure.
The method and the device solve the problem of inconsistent data caused by time difference in the synchronization process of the master database and the slave database, and can rapidly switch the master database and the slave database, thereby greatly shortening the abnormal recovery time of the Redis database, ensuring that a user hardly feels when using the Redis database, and greatly improving the usability of the Redis database cluster.
After the step 207 is completed, the step of obtaining the operation logs of the master database in the Redis database cluster and distributing the operation logs to the slave databases in the Redis database cluster is returned to be executed.
And monitoring the newly set master database again by implementing the monitoring component, and assisting in carrying out a data synchronization processing process between the master database and each slave database.
Further, after identifying the target slave database as a new master database, the method further comprises:
upon detecting that the failure of the historical master database is repaired, identifying the historical master database as a slave database.
Correspondingly, after the historical master database is identified as the slave database, the method further comprises the following steps: acquiring a second operation log synchronization request sent by a history master database; determining a fifth operation log which is finally obtained from the historical master database before the historical master database fails according to the second operation log synchronization request; according to the fifth operation log, acquiring a second target operation log of which the distribution sequence is between the fifth operation log and the operation log which is currently and latest distributed to each slave database; and distributing the second target operation log to a historical master database.
And after the real-time monitoring component monitors that the historical master database component is recovered, the real-time monitoring component automatically identifies the historical master database component as a slave database, and distributes new logs of the master database to the historical master database for synchronous operation.
Specifically, the real-time monitoring component acquires operation logs generated between the last executed operation log of the historical master database and the latest operation log acquired by the real-time monitoring component from the new master database, distributes the operation logs to the historical master database identified as the slave database, and the historical master database analyzes the operation logs and executes corresponding operations, so that data consistency between the historical master database and the new master database is realized.
And after the historical database component is repaired and restarted, calling a fresh () interface of the real-time monitoring component to acquire log data between the last executed topocid of the real-time monitoring component and the latest topocid of the real-time monitoring component, analyzing and sequentially executing the log data so as to ensure that the data of the real-time monitoring component is consistent with the data of the new main database.
The process can greatly shorten the abnormal recovery time of the Redis database, and enables users to feel almost nothing when using the Redis database, thereby greatly improving the usability of the Redis database.
In the embodiment of the application, operation logs of a master database in a Redis database cluster are obtained and distributed to each slave database in the Redis database cluster in sequence, when a first operation log which is newly distributed by a first slave database is not continuous with a second operation log which is distributed last, a first operation log synchronization request sent by the first slave database is obtained, according to the first operation log synchronization request, a first target operation log with the distribution sequence between the first operation log and the second operation log is obtained, the first target operation log is distributed to the first slave database, when the master database is monitored to have a fault, the master database is identified as a historical master database, and the target slave database is identified as a new master database. The process realizes the real-time monitoring of the master database by introducing a real-time monitoring component, finds out missing operation logs according to the distribution sequence of the operation logs when the operation logs distributed latest in the slave database are discontinuous with the operation logs stored latest locally, and distributes the missing operation logs to the slave database, thereby ensuring the consistency of the data in the slave database and the data in the master database, solving the problem of inconsistent data caused by time difference in the synchronization process of the master database and the slave database, and being capable of rapidly switching the master database and the slave database, thereby greatly shortening the abnormal recovery time of the Redis database, leading a user to feel almost no sense when using the Redis database, and greatly improving the usability of the Redis.
Referring to fig. 3, fig. 3 is a block diagram of a data processing apparatus based on a database cluster according to an embodiment of the present application, and for convenience of description, only a part related to the embodiment of the present application is shown.
The database cluster-based data processing apparatus 300 includes:
a first obtaining module 301, configured to obtain an operation log of a master database in a Redis database cluster, and sequentially distribute the operation log to each slave database in the Redis database cluster;
a second obtaining module 302, configured to obtain a first oplog synchronization request sent by a first slave database when a first oplog that is newly distributed by the first slave database is not consecutive to a second oplog that is distributed last;
a third obtaining module 303, configured to obtain, according to the first operation log synchronization request, a first target operation log whose distribution order is located between the first operation log and the second operation log;
a distributing module 304, configured to distribute the first target operation log to the first slave database.
The first obtaining module is specifically configured to:
acquiring an operation log of a master database in a Redis database cluster;
carrying out serial number marking on the operation logs according to the acquisition sequence to obtain log numbers;
and sequentially distributing the operation logs containing the log numbers to each slave database in the Redis database cluster.
The first operation log synchronization request carries a first log number of the first operation log and a second log number of the second operation log;
wherein the first log number is not consecutive to the second log number.
The third obtaining module 303 is specifically configured to:
and according to the first operation log synchronization request, acquiring a first target operation log with a log number between the first log number and the second log number.
The device also includes: a database identification module to:
when the master database is monitored to have faults, determining a third operation log which is distributed to each slave database at last, and sequentially accessing a fourth operation log which is distributed to each slave database at last;
determining a target slave database of which the fourth operation log is consistent with the third operation log from the slave database according to the third operation log and the fourth operation log;
identifying the master database as a historical master database and the target slave database as a new master database;
and returning and executing the operation log for acquiring the master database in the Redis database cluster, and distributing the operation log to each slave database in the Redis database cluster in sequence.
Wherein, the database identification module is further configured to:
identifying the historical master database as a slave database upon detecting that a failure of the historical master database is repaired.
The device also includes: a data synchronization module to:
acquiring a second operation log synchronization request sent by the historical master database;
determining a fifth operation log which is obtained from the historical master database last before the historical master database fails according to the second operation log synchronization request;
according to the fifth operation log, acquiring a second target operation log of which the distribution sequence is positioned between the fifth operation log and the operation log which is currently and latest distributed to each slave database;
distributing the second target oplog to the historical master database.
The data processing device based on the database cluster provided by the embodiment of the application can realize each process of the data processing method based on the database cluster, can achieve the same technical effect, and is not repeated here to avoid repetition.
Fig. 4 is a block diagram of an electronic device according to an embodiment of the present application. As shown in the figure, the electronic apparatus 4 of the embodiment includes: at least one processor 40 (only one shown in fig. 4), a memory 41, and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, the steps of any of the various method embodiments described above being implemented when the computer program 42 is executed by the processor 40.
The electronic device 4 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The electronic device 4 may include, but is not limited to, a processor 40, a memory 41. Those skilled in the art will appreciate that fig. 4 is merely an example of an electronic device 4 and does not constitute a limitation of the electronic device 4 and may include more or fewer components than shown, or some components may be combined, or different components, e.g., the electronic device may also include input-output devices, network access devices, buses, etc.
The Processor 40 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or a memory of the electronic device 4. The memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 41 is used for storing the computer program and other programs and data required by the electronic device. The memory 41 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/electronic device and method may be implemented in other ways. For example, the above-described apparatus/electronic device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a computer-readable storage medium and can realize the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
The present application realizes all or part of the processes in the methods of the embodiments described above, and may also be implemented by a computer program product, when the computer program product runs on an electronic device, the electronic device is enabled to implement the steps in the embodiments of the methods described above when executed.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A data processing method based on database cluster is characterized by comprising the following steps:
acquiring an operation log of a master database in a Redis database cluster, and sequentially distributing the operation log to each slave database in the Redis database cluster;
when a first operation log which is distributed by a first slave database latest is not continuous with a second operation log which is distributed last, a first operation log synchronization request sent by the first slave database is obtained;
according to the first operation log synchronization request, acquiring a first target operation log with a distribution sequence between the first operation log and the second operation log;
distributing the first target oplog to the first slave database.
2. The data processing method according to claim 1, wherein the obtaining an operation log of a master database in a Redis database cluster, and sequentially distributing the operation log to each slave database in the Redis database cluster comprises:
acquiring an operation log of a master database in a Redis database cluster;
carrying out serial number marking on the operation logs according to the acquisition sequence to obtain log numbers;
and sequentially distributing the operation logs containing the log numbers to each slave database in the Redis database cluster.
3. The data processing method according to claim 2, wherein the first oplog synchronization request carries a first log number of the first oplog and a second log number of the second oplog;
wherein the first log number is not consecutive to the second log number.
4. The data processing method according to claim 3, wherein the obtaining, according to the first oplog synchronization request, a first target oplog having a distribution order between the first oplog and the second oplog comprises:
and according to the first operation log synchronization request, acquiring a first target operation log with a log number between the first log number and the second log number.
5. The data processing method of claim 1, wherein after the distributing the first target oplog to the first slave database, further comprising:
when the master database is monitored to have faults, determining a third operation log which is distributed to each slave database at last, and sequentially accessing a fourth operation log which is distributed to each slave database at last;
determining a target slave database of which the fourth operation log is consistent with the third operation log from the slave database according to the third operation log and the fourth operation log;
identifying the master database as a historical master database and the target slave database as a new master database;
and returning and executing the operation log for acquiring the master database in the Redis database cluster, and distributing the operation log to each slave database in the Redis database cluster in sequence.
6. The data processing method of claim 5, wherein after identifying the target slave database as a new master database, further comprising:
identifying the historical master database as a slave database upon detecting that a failure of the historical master database is repaired.
7. The data processing method of claim 5, wherein after identifying the historical master database as a slave database, further comprising:
acquiring a second operation log synchronization request sent by the historical master database;
determining a fifth operation log which is obtained from the historical master database last before the historical master database fails according to the second operation log synchronization request;
according to the fifth operation log, acquiring a second target operation log of which the distribution sequence is positioned between the fifth operation log and the operation log which is currently and latest distributed to each slave database;
distributing the second target oplog to the historical master database.
8. A database cluster-based data processing apparatus, comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring an operation log of a master database in a Redis database cluster and sequentially distributing the operation log to each slave database in the Redis database cluster;
the second acquisition module is used for acquiring a first operation log synchronization request sent by a first slave database when a first operation log which is distributed latest by the first slave database is discontinuous with a second operation log which is distributed last time;
a third obtaining module, configured to obtain, according to the first operation log synchronization request, a first target operation log whose distribution order is located between the first operation log and the second operation log;
a distribution module, configured to distribute the first target operation log to the first slave database.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the steps of the method according to any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011588369.0A CN112612853A (en) | 2020-12-28 | 2020-12-28 | Data processing method and device based on database cluster and electronic equipment |
| PCT/CN2021/127181 WO2022142665A1 (en) | 2020-12-28 | 2021-10-28 | Database cluster-based data processing method and apparatus, and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011588369.0A CN112612853A (en) | 2020-12-28 | 2020-12-28 | Data processing method and device based on database cluster and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112612853A true CN112612853A (en) | 2021-04-06 |
Family
ID=75248592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011588369.0A Pending CN112612853A (en) | 2020-12-28 | 2020-12-28 | Data processing method and device based on database cluster and electronic equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112612853A (en) |
| WO (1) | WO2022142665A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177093A (en) * | 2021-05-24 | 2021-07-27 | 商汤国际私人有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113806301A (en) * | 2021-09-29 | 2021-12-17 | 中国平安人寿保险股份有限公司 | Data synchronization method, device, server and storage medium |
| WO2022142665A1 (en) * | 2020-12-28 | 2022-07-07 | 深圳壹账通智能科技有限公司 | Database cluster-based data processing method and apparatus, and electronic device |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115438087B (en) * | 2022-11-10 | 2023-03-24 | 广州思迈特软件有限公司 | Data query method, device, storage medium and equipment based on cache library |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7457829B2 (en) * | 2003-06-23 | 2008-11-25 | Microsoft Corporation | Resynchronization of multiple copies of a database after a divergence in transaction history |
| CN101751394B (en) * | 2008-12-16 | 2011-11-16 | 青岛海信传媒网络技术有限公司 | Method and system for synchronizing data |
| CN108984569A (en) * | 2017-06-05 | 2018-12-11 | 中兴通讯股份有限公司 | Database switching method, system and computer readable storage medium |
| CN110019062A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method of data synchronization and system |
| CN112612853A (en) * | 2020-12-28 | 2021-04-06 | 深圳壹账通智能科技有限公司 | Data processing method and device based on database cluster and electronic equipment |
-
2020
- 2020-12-28 CN CN202011588369.0A patent/CN112612853A/en active Pending
-
2021
- 2021-10-28 WO PCT/CN2021/127181 patent/WO2022142665A1/en active Application Filing
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022142665A1 (en) * | 2020-12-28 | 2022-07-07 | 深圳壹账通智能科技有限公司 | Database cluster-based data processing method and apparatus, and electronic device |
| CN113177093A (en) * | 2021-05-24 | 2021-07-27 | 商汤国际私人有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113806301A (en) * | 2021-09-29 | 2021-12-17 | 中国平安人寿保险股份有限公司 | Data synchronization method, device, server and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022142665A1 (en) | 2022-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112612853A (en) | Data processing method and device based on database cluster and electronic equipment | |
| CN109471626B (en) | Page logic structure, page generation method, page data processing method and device | |
| CN110162429A (en) | System repair, server and storage medium | |
| CN108717516B (en) | File labeling method, terminal and medium | |
| CN109783321B (en) | Monitoring data management method and device and terminal equipment | |
| CN106775445B (en) | Data migration method and device | |
| CN111475334A (en) | TiDB database maintenance method and device, computer equipment and storage medium | |
| CN117270891A (en) | Firmware burning method, device, terminal and storage medium of Bluetooth headset | |
| CN116208487A (en) | Method, device, equipment and medium for upgrading consensus algorithm in block chain system | |
| CN115640280A (en) | Data migration method and device | |
| CN111984723A (en) | Data synchronization method and device and terminal equipment | |
| CN115470189A (en) | Data query method and device and readable storage medium | |
| CN111538520B (en) | Updating method and device for super-converged cluster, terminal and storage medium | |
| CN114385657A (en) | Data storage method, device and storage medium | |
| CN113110846A (en) | Method and device for acquiring environment variable | |
| CN113190531A (en) | Database migration method, device, equipment and storage medium | |
| CN108632366B (en) | File downloading method and device and terminal equipment | |
| CN112860376A (en) | Snapshot chain making method and device, electronic equipment and storage medium | |
| US20150046414A1 (en) | Computer product, managing apparatus, and managing method | |
| CN112015459A (en) | Method and device for upgrading image file, terminal and readable storage medium | |
| CN114547184A (en) | Personnel information synchronization method, terminal device and storage medium | |
| CN112787849B (en) | Block chain state control method and device, terminal and storage medium | |
| CN110321340B (en) | Database updating method, database updating device and mobile terminal | |
| CN110008706B (en) | Host security state management method and device and terminal equipment | |
| CN111221817B (en) | Service information data storage method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40049932 Country of ref document: HK |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |