WO2022142665A1 - 基于数据库集群的数据处理方法、装置及电子设备 - Google Patents
基于数据库集群的数据处理方法、装置及电子设备 Download PDFInfo
- Publication number
- WO2022142665A1 WO2022142665A1 PCT/CN2021/127181 CN2021127181W WO2022142665A1 WO 2022142665 A1 WO2022142665 A1 WO 2022142665A1 CN 2021127181 W CN2021127181 W CN 2021127181W WO 2022142665 A1 WO2022142665 A1 WO 2022142665A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- operation log
- database
- log
- slave
- cluster
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 46
- 238000012544 monitoring process Methods 0.000 claims description 37
- 238000004590 computer program Methods 0.000 claims description 26
- 238000012545 processing Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 description 25
- 230000006870 function Effects 0.000 description 9
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0778—Dumping, i.e. gathering error/state information after a fault for later diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
Definitions
- the present application belongs to the technical field of cloud storage, and in particular, relates to a data processing method, apparatus and electronic device based on a database cluster.
- the Remote Dictionary Server (Redis) database cluster has gradually replaced the original cache system due to its fast read and write speed and stable performance, and is widely used by various systems.
- a Redis database cluster is divided into a master database and a slave database.
- the data in the slave database needs to be consistent with the data in the master database to ensure the normal operation of read and write functions in the Redis database cluster.
- the slave database that can communicate normally is used as the master database, but there is a problem in the synchronization process of the master and slave databases. The time difference results in inconsistent data and cannot meet normal business requirements.
- the embodiments of the present application provide a data processing method, device and electronic device based on a database cluster, so as to solve the problem that in the prior art, when the Redis master database is down, it takes a long time to restore the service, and the master-slave database synchronization process takes a long time. There is a problem that the time difference leads to data inconsistency.
- a first aspect of the embodiments of the present application provides a data processing method based on a database cluster, including:
- a second aspect of the embodiments of the present application provides a data processing apparatus based on a database cluster, including:
- the first obtaining module is used to obtain the operation log of the master database in the Redis database cluster, and sequentially distribute the operation log to each slave database in the Redis database cluster;
- a second obtaining module configured to obtain the first operation log synchronization request sent by the first slave database when the first operation log newly distributed by the first slave database is not continuous with the second operation log distributed last;
- a third acquisition module configured to acquire, according to the first operation log synchronization request, a first target operation log whose distribution order is between the first operation log and the second operation log;
- a distribution module configured to distribute the first target operation log to the first slave database.
- a third aspect of the embodiments of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program The steps of the method as described in the first aspect are implemented.
- a fourth aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
- a fifth aspect of the present application provides a computer program product, which, when the computer program product runs on an electronic device, causes the electronic device to perform the steps of the method in the first aspect.
- the operation log is distributed to each slave database in the Redis database cluster in turn, and the first operation log distributed by the first slave database is the latest.
- obtain the first operation log synchronization request sent first from the database and according to the first operation log synchronization request, obtain the distribution sequence between the first operation log and the second operation log.
- the first target operation log between the logs distributes the first target operation log to the first slave database.
- the missing operation log is found according to the distribution order of the operation log, and Distribute it to the slave database to ensure that the data in the slave database is consistent with the data in the master database.
- FIG. 1 is a flowchart 1 of a data processing method based on a database cluster provided by an embodiment of the present application;
- FIG. 2 is a second flowchart of a data processing method based on a database cluster provided by an embodiment of the present application
- FIG. 3 is a structural diagram of a data processing apparatus based on a database cluster provided by an embodiment of the present application
- FIG. 4 is a structural diagram of an electronic device provided by an embodiment of the present application.
- the term “if” may be contextually interpreted as “when” or “once” or “in response to determining” or “in response to detecting” .
- the phrases “if it is determined” or “if the [described condition or event] is detected” may be interpreted, depending on the context, to mean “once it is determined” or “in response to the determination” or “once the [described condition or event] is detected. ]” or “in response to detection of the [described condition or event]”.
- the electronic devices described in the embodiments of the present application include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touch pads).
- the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (eg, a touch screen display and/or a touch pad).
- electronic devices including displays and touch-sensitive surfaces are described. It should be understood, however, that the electronic device may include one or more other physical user interface devices such as a physical keyboard, mouse, and/or joystick.
- Electronic devices support a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk burning applications, spreadsheet applications, gaming applications, telephony Apps, Video Conferencing Apps, Email Apps, Instant Messaging Apps, Workout Support Apps, Photo Management Apps, Digital Camera Apps, Digital Video Camera Apps, Web Browsing Apps, Digital Music Player Apps and/or digital video player applications.
- applications such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk burning applications, spreadsheet applications, gaming applications, telephony Apps, Video Conferencing Apps, Email Apps, Instant Messaging Apps, Workout Support Apps, Photo Management Apps, Digital Camera Apps, Digital Video Camera Apps, Web Browsing Apps, Digital Music Player Apps and/or digital video player applications.
- Various applications that may be executed on an electronic device may use at least one common physical user interface device, such as a touch-sensitive surface.
- a touch-sensitive surface One or more functions of the touch-sensitive surface and corresponding information displayed on the electronic device may be adjusted and/or changed between applications and/or within respective applications.
- the common physical architecture of the electronic device eg, touch sensitive surfaces
- FIG. 1 is a flowchart 1 of a data processing method based on a database cluster provided by an embodiment of the present application.
- the data processing method based on the database cluster is specifically realized by implementing a monitoring component, or by other designed hardware or software.
- a data processing method based on database cluster includes the following steps:
- Step 101 Obtain the operation log of the master database in the Redis database cluster, and distribute the operation log to each slave database in the Redis database cluster in turn.
- the database is divided into two categories: the master database (master) and the slave database (slave).
- master can have multiple slaves, but a slave can only correspond to one master.
- the data of the slave database is synchronized, specifically, through the operation log obtained by the real-time monitoring component from the master database.
- the operation log is transmitted between the master database and the slave database, so that the slave database can update its own data synchronously based on the operation log of the master database.
- the operation log records the data update operation of the master database itself.
- the data update operation includes but is not limited to data deletion, Data creation, data modification, establishment of relationship between data, data write operation, creation of data table, import of data table, etc.
- the real-time monitoring component after the real-time monitoring component obtains the operation log, it can save it locally. Specifically, it can delete the operation log saved locally after a set period of time, or the number of the retained operation log can be maintained at a set value. , and if it exceeds the set value, the operation log obtained first will be deleted.
- the acquisition of the operation log of the main database in the Redis database cluster is to acquire according to a preset frequency, or to acquire when it is detected that a new operation log is generated in the main database, or when it is detected that data occurs in the main database During the update operation, whenever these data operations are monitored, the operation log in the main database needs to be obtained once.
- the operation log of the master database in the Redis database cluster is obtained, and the operation log is distributed to each slave database in the Redis database cluster in turn, including:
- the log number is the number sorted according to the order in which the operation logs are obtained.
- the log numbers are in one-to-one correspondence with the order in which the operation logs are obtained.
- the real-time monitoring component When the real-time monitoring component obtains the operation log of the master database, it needs to mark the obtained operation log with a serial number.
- the acquisition order is distributed sequentially.
- the log function needs to be enabled in the main database, and a unique sequential number topicid is generated for each operation log of the main database obtained by the real-time monitoring component.
- the log function also needs to be turned on from the database.
- the real-time monitoring component is configured with the IPs of the master library and the slave library to realize information transfer between the master library and the slave library, so as to extract the log of the master database in real time and distribute it to the slave database.
- Step 102 when the first operation log newly distributed by the first slave database is not continuous with the second operation log distributed last time, obtain a first operation log synchronization request sent by the first slave database.
- the first slave database When the first slave database receives the currently newly distributed first operation log, it has not stored the first operation log locally, and needs to combine the newly distributed first operation log with the last distributed second operation log. The relationship between the logs is judged to determine whether the two are continuous operation logs. The first slave database stores the first operation log locally when it is determined that the two are continuous logs, and executes the first operation log to achieve data consistency with the master database.
- the slave database can determine whether the operation log is a continuous log by comparing the log numbers of the operation logs distributed by the real-time monitoring component to determine whether the log number is a continuous number. It is judged whether the log number of the locally stored second operation log and the log number of the first operation log are consecutive numbers, and if so, it is determined that the two operation logs are consecutive logs, and vice versa.
- the topicid of the database is t-001
- the topicid of the first operation log pushed by the real-time monitoring component is t-003
- the operation log with the topicid of t-002 is selected, the currently received first operation log and the latest operation log stored locally may not be continuous logs.
- the reason for the discontinuous operation log may be data loss caused by network interruption, or after the original master database has been repaired and restarted to become the slave database, the topicid recorded by itself is no longer up-to-date, resulting in data loss.
- the first operation log synchronization request carries a first log number of the first operation log and a second log number of the second operation log; wherein the first log number and the second log number are Log numbers are not consecutive.
- the sequence label of the operation log can be distributed along with it.
- Step 103 Acquire, according to the first operation log synchronization request, a first target operation log whose distribution order is between the first operation log and the second operation log.
- the log number of the first target operation log whose distribution order is located between the first operation log and the second operation log is between the log number of the first operation log and the log number of the second operation log.
- obtaining the first target operation log whose distribution order is located between the first operation log and the second operation log includes:
- a first target operation log whose log number is between the first log number and the second log number is obtained.
- the real-time monitoring component obtains the first target operation log whose distribution order is between the first operation log and the second operation log, that is, the log whose log number is between the first operation log and the second operation log
- the target operation logs between the numbers are distributed to the first slave database.
- the first slave database parses and executes the log data in sequence.
- the topicid of the latest operation log currently stored locally from the database is t-003
- the topicid of the last operation log sent from the real-time monitoring component is already t-005.
- the slave database parses the log data in order for synchronization operations, and records the topicid of the last operation log. , to ensure that the data stored in the slave database is consistent with the data in the master database.
- the slave database will also update the topicid of the latest operation log after each log synchronization operation is performed, which is used to ensure that the data of itself and the master database are consistent based on the log number.
- Step 104 Distribute the first target operation log to the first slave database.
- the implementation finds the missing operation log and distributes it to the slave database to ensure the consistency of the data in the slave database with the data in the master database.
- the operation log of the master database in the Redis database cluster is obtained, and the operation log is distributed to each slave database in the Redis database cluster in turn.
- the distributed second operation log is discontinuous, obtain the first operation log synchronization request sent from the first database, and obtain the distribution order between the first operation log and the second operation log according to the first operation log synchronization request.
- the first target operation log which distributes the first target operation log to the first slave database.
- the real-time monitoring of the main database is realized by introducing a real-time monitoring component.
- the missing operation log is found according to the distribution order of the operation log, and Distribute it to the slave database to ensure that the data in the slave database is consistent with the data in the master database.
- the operation log is distributed to each slave database in the redis database cluster in turn, and the newly distributed first operation log of the first slave database and the locally stored
- the latest operation log is discontinuous, obtain the first operation log synchronization request sent from the database first, and obtain the first target operation whose distribution order is between the first operation log and the latest operation log according to the first operation log synchronization request log, and distribute the first target operation log to the first slave database.
- the real-time monitoring of the main database is realized by introducing a real-time monitoring component.
- the missing operation log is found according to the distribution order of the operation log, and Distribute it to the slave database to ensure that the data in the slave database is consistent with the data in the master database.
- FIG. 2 is a second flowchart of a data processing method based on a database cluster provided by an embodiment of the present application. As shown in Figure 2, a data processing method based on database cluster, the method includes the following steps:
- Step 201 Obtain the operation log of the master database in the Redis database cluster, and distribute the operation log to each slave database in the Redis database cluster in turn.
- Step 202 when the first operation log newly distributed by the first slave database is discontinuous with the second operation log distributed last time, obtain the first operation log synchronization request sent by the first slave database.
- step 102 The implementation process of this step is the same as the implementation process of step 102 in the foregoing embodiment, and details are not repeated here.
- Step 203 according to the first operation log synchronization request, obtain a first target operation log whose distribution order is between the first operation log and the second operation log.
- step 103 The implementation process of this step is the same as the implementation process of step 103 in the foregoing embodiment, and is not repeated here.
- Step 204 Distribute the first target operation log to the first slave database.
- step 104 The implementation process of this step is the same as the implementation process of step 104 in the foregoing embodiment, and details are not repeated here.
- Step 205 when monitoring the failure of the master database, determine the third operation log distributed to each slave database at last, and sequentially access the fourth operation log distributed last in each slave database.
- Step 206 determine from the slave database a target slave database whose fourth operation log is consistent with the third operation log.
- the slave databases in the Redis database cluster are cyclically accessed in turn, and the latest operation log recorded in each slave database (that is, the fourth operation log distributed last) and the latest operation log in the real-time monitoring component are judged. Whether the operation log (that is, the third operation log distributed to each slave database at the end) is consistent, if it is consistent, set the current slave database as the new master database.
- Step 207 identifying the master database as a historical master database, and identifying the target slave database as a new master database.
- the real-time monitoring component monitors whether the main database component works normally in real time.
- the real-time monitoring component monitors that the master database is down in real time, it immediately pings all configured database IPs from the configuration file, and compares the topicid of the slave library with its own topicid. If the topicid is the same, set the current slave library as the new one. main library.
- the reason for the inconsistency may be that the original faulty slave library has been restored to use, or the slave library has not received the latest operation log due to a network failure.
- This solution solves the problem of data inconsistency caused by the time difference between the master and slave databases, and can quickly switch the master and slave databases, thereby greatly shortening the abnormal recovery time of the Redis database, making users almost indifferent when using it, and greatly improving the Redis database cluster. availability.
- step 207 After the execution of step 207 is completed, return to the steps of obtaining the operation log of the master database in the Redis database cluster, and sequentially distributing the operation log to each slave database in the Redis database cluster.
- the target slave database after identifying the target slave database as the new master database, it also includes:
- the historical master database After detecting that the failure of the historical master database is repaired, the historical master database is identified as a slave database.
- the method further includes: obtaining a second operation log synchronization request sent by the historical master database; The fifth operation log obtained last in the database; according to the fifth operation log, obtain the second target operation log whose distribution order is between the fifth operation log and the operation log currently distributed to each slave database; the second target operation log is obtained; Operational logs are distributed to the historical master database.
- the real-time monitoring component monitors the recovery of the historical master database component, it automatically identifies it as a slave database, and distributes the logs of the new master database to the historical master database for synchronization operations.
- the real-time monitoring component obtains the operation log generated between the last operation log of the historical master database and the latest operation log obtained by the real-time monitoring component from the new master database, and distributes it to the historical master database that has been identified as the slave database.
- the historical master database parses the operation logs, and executes corresponding operations to achieve data consistency between the historical master database and the new master database.
- call the fresh() interface of the real-time monitoring component to obtain the log data between the last executed topicid of the real-time monitoring component and the latest topicid of the real-time monitoring component, parse and execute the log data in sequence, so as to ensure that its own data is compatible with the new main database. Data remains consistent.
- This process can greatly shorten the abnormal recovery time of the Redis database, make users feel almost indifferent when using it, and greatly improve the availability of Redis.
- the operation log of the master database in the Redis database cluster is obtained, and the operation log is distributed to each slave database in the Redis database cluster in turn.
- the distributed second operation log is discontinuous, obtain the first operation log synchronization request sent from the first database, and obtain the distribution order between the first operation log and the second operation log according to the first operation log synchronization request.
- the first target operation log is distributed to the first slave database, and when the failure of the master database is monitored, the master database is identified as the historical master database, and the target slave database is identified as the new master database.
- the missing operation log is found according to the distribution order of the operation log, and Distribute it to the slave database to ensure the consistency of the data in the slave database and the data in the master database, solve the problem of data inconsistency due to the time difference in the synchronization process of the master and slave databases, and can quickly switch the master and slave databases to greatly shorten the time.
- the abnormal recovery time of the Redis database makes users almost indifferent when using it, which greatly improves the availability of Redis.
- FIG. 3 is a structural diagram of a data processing apparatus based on a database cluster provided by an embodiment of the present application. For convenience of description, only parts related to the embodiment of the present application are shown.
- the database cluster-based data processing apparatus 300 includes:
- the first obtaining module 301 is used for obtaining the operation log of the master database in the Redis database cluster, and sequentially distributing the operation log to each slave database in the Redis database cluster;
- the second obtaining module 302 is configured to obtain the first operation log synchronization request sent by the first slave database when the first operation log newly distributed by the first slave database is not continuous with the second operation log distributed last time ;
- the third obtaining module 303 is configured to obtain, according to the first operation log synchronization request, the first target operation log whose distribution order is between the first operation log and the second operation log;
- the distribution module 304 is configured to distribute the first target operation log to the first slave database.
- the first acquisition module is specifically used for:
- the first operation log synchronization request carries a first log number of the first operation log and a second log number of the second operation log;
- the first log number and the second log number are not consecutive.
- the third acquisition module 303 is specifically used for:
- a first target operation log whose log number is between the first log number and the second log number is obtained.
- the device also includes: a database identification module for:
- the third operation log and the fourth operation log determine, from the slave database, a target slave database whose fourth operation log is consistent with the third operation log;
- the database identification module is also used for:
- the historical master database After detecting that the failure of the historical master database is repaired, the historical master database is identified as a slave database.
- the device also includes: a data synchronization module for:
- the fifth operation log obtain the second target operation log whose distribution order is located between the fifth operation log and the operation log currently distributed to each of the slave databases;
- the database cluster-based data processing apparatus provided by the embodiments of the present application can implement each process of the above embodiments of the database cluster-based data processing method, and can achieve the same technical effect. To avoid repetition, details are not described here.
- FIG. 4 is a structural diagram of an electronic device provided by an embodiment of the present application.
- the electronic device 4 of this embodiment includes: at least one processor 40 (only one is shown in FIG. 4 ), a memory 41 , and a memory 41 stored in the memory 41 and available in the at least one processor 40
- the computer program 42 running on the processor 40 implements the steps in any of the foregoing method embodiments when the processor 40 executes the computer program 42 .
- the electronic device 4 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server.
- the electronic device 4 may include, but is not limited to, a processor 40 and a memory 41 .
- FIG. 4 is only an example of the electronic device 4, and does not constitute a limitation to the electronic device 4, and may include more or less components than the one shown, or combine some components, or different components
- the electronic device may further include an input and output device, a network access device, a bus, and the like.
- the processor 40 may be a central processing unit (Central Processing Unit, CPU), other general-purpose processors, Digital Signal Processor (Digital Signal Processor, DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array) Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
- CPU Central Processing Unit
- DSP Digital Signal Processor
- ASIC Application Specific Integrated Circuit
- Field-Programmable Gate Array Field-Programmable Gate Array
- FPGA Field-Programmable Gate Array
- a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
- the memory 41 may be an internal storage unit of the electronic device 4 , such as a hard disk or a memory of the electronic device 4 .
- the memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk equipped on the electronic device 4, a smart memory card (Smart memory card). Media Card, SMC), Secure Digital (Secure Digital, SD) card, flash memory card (Flash Card), etc.
- the memory 41 may also include both an internal storage unit of the electronic device 4 and an external storage device.
- the memory 41 is used to store the computer program and other programs and data required by the electronic device.
- the memory 41 can also be used to temporarily store data that has been output or will be output.
- the disclosed apparatus/electronic device and method may be implemented in other manners.
- the above-described embodiments of the apparatus/electronic device are only illustrative.
- the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units. Or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
- the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.
- the units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
- the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
- the integrated modules/units if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium.
- the present application can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing the relevant hardware through a computer program.
- the computer program can be stored in a computer-readable storage medium, and the computer When the program is executed by the processor, the steps of the foregoing method embodiments can be implemented.
- the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like.
- the computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable media Electric carrier signals and telecommunication signals are not included.
- the computer-readable storage medium may be non-volatile or volatile.
- This application implements all or part of the processes in the methods of the above embodiments, and can also be implemented by a computer program product.
- the computer program product runs on an electronic device, so that the electronic device can implement the above methods and embodiments when executed.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请适用于云存储技术领域，提供一种基于数据库集群的数据处理方法、装置及电子设备，其中方法包括：获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库；在第一从数据库最新被分发的第一操作日志与本地存储的最新操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求；根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述最新操作日志之间的第一目标操作日志；将所述第一目标操作日志分发至所述第一从数据库。该方案能够确保从数据库中数据与主数据库中数据的一致性。
Description
本申请要求于2020年12月28日提交中国专利局，申请号为202011588369.0，发明名称为“基于数据库集群的数据处理方法、装置及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
本申请属于云存储技术领域，尤其涉及一种基于数据库集群的数据处理方法、装置及电子设备。
远程字典服务（Remote Dictionary Server，Redis）数据库集群由于读写速度快、性能稳定，已经逐渐取代原有缓存系统，被各个系统广泛使用。
Redis数据库集群中，分为主数据库与从数据库，从数据库中的数据需要与主数据库中数据保持一致，以确保Redis数据库集群中读写等功能的正常运行。
目前遇到异常情况，例如Redis中主数据库宕机或者主从数据库间通信出现中断等，则会造成主从库数据不一致或长时间Redis数据库不可用的问题。
发明人发现，对于目前常规解决方案：1、需要人工介入进行数据库故障修复与重启，但恢复服务耗时长；2、启用能够正常通信的从数据库作为主数据库，但是主、从数据库同步过程中存在时间差，导致数据不一致，不能满足正常业务诉求。
有鉴于此，本申请实施例提供了一种基于数据库集群的数据处理方法、装置及电子设备，以解决现有技术中Redis主数据库出现宕机时恢复服务耗时长，且主从数据库同步过程中存在时间差导致数据不一致的问题。
本申请实施例的第一方面提供了一种基于数据库集群的数据处理方法，包括：
获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库；
在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求；
根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述第二操作日志之间的第一目标操作日志；
将所述第一目标操作日志分发至所述第一从数据库。
本申请实施例的第二方面提供了一种基于数据库集群的数据处理装置，包括：
第一获取模块，用于获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库；
第二获取模块，用于在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求；
第三获取模块，用于根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述第二操作日志之间的第一目标操作日志；
分发模块，用于将所述第一目标操作日志分发至所述第一从数据库。
本申请实施例的第三方面提供了一种电子设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如第一方面所述方法的步骤。
本申请实施例的第四方面提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现如第一方面所述方法的步骤。
本申请的第五方面提供了一种计算机程序产品，当所述计算机程序产品在电子设备上运行时，使得所述电子设备执行上述第一方面所述方法的步骤。
由上可见，本申请实施例中，通过获取Redis数据库集群中主数据库的操作日志，依次将操作日志分发至Redis数据库集群中的各从数据库，在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取第一从数据库发送的第一操作日志同步请求，根据该第一操作日志同步请求，获取分发次序位于第一操作日志与该第二操作日志之间的第一目标操作日志，将第一目标操作日志分发至第一从数据库。该过程通过引入实时监控组件实现对主数据库的实时监控，在从数据库中最新被分发的操作日志与本地存储的最新操作日志不连续时，依照操作日志的分发次序找出缺失的操作日志，并将其分发至从数据库中，确保从数据库中数据与主数据库中数据的一致性。
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种基于数据库集群的数据处理方法的流程图一；
图2是本申请实施例提供的一种基于数据库集群的数据处理方法的流程图二；
图3是本申请实施例提供的一种基于数据库集群的数据处理装置的结构图；
图4是本申请实施例提供的一种电子设备的结构图。
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、技术之类的具体细节，以便透彻理解本申请实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本申请。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本申请的描述。
应当理解，当在本说明书和所附权利要求书中使用时，术语“包括”指示所描述特征、整体、步骤、操作、元素和/或组件的存在，但并不排除一个或多个其它特征、整体、步骤、操作、元素、组件和/或其集合的存在或添加。
还应当理解，在此本申请说明书中所使用的术语仅仅是出于描述特定实施例的目的而并不意在限制本申请。如在本申请说明书和所附权利要求书中所使用的那样，除非上下文清楚地指明其它情况，否则单数形式的“一”、“一个”及“该”意在包括复数形式。
还应当进一步理解，在本申请说明书和所附权利要求书中使用的术语“和/或”是指相关联列出的项中的一个或多个的任何组合以及所有可能组合，并且包括这些组合。
如在本说明书和所附权利要求书中所使用的那样，术语“如果”可以依据上下文被解释为“当...时”或“一旦”或“响应于确定”或“响应于检测到”。类似地，短语“如果确定”或“如果检测到[所描述条件或事件]”可以依据上下文被解释为意指“一旦确定”或“响应于确定”或“一旦检测到[所描述条件或事件]”或“响应于检测到[所描述条件或事件]”。
具体实现中，本申请实施例中描述的电子设备包括但不限于诸如具有触摸敏感表面(例如，触摸屏显示器和/或触摸板)的移动电话、膝上型计算机或平板计算机之类的其它便携式设备。还应当理解的是，在某些实施例中，所述设备并非便携式通信设备，而是具有触摸敏感表面(例如，触摸屏显示器和/或触摸板)的台式计算机。
在接下来的讨论中，描述了包括显示器和触摸敏感表面的电子设备。然而，应当理解的是，电子设备可以包括诸如物理键盘、鼠标和/或控制杆的一个或多个其它物理用户接口设备。
电子设备支持各种应用程序，例如以下中的一个或多个：绘图应用程序、演示应用程序、文字处理应用程序、网站创建应用程序、盘刻录应用程序、电子表格应用程序、游戏应用程序、电话应用程序、视频会议应用程序、电子邮件应用程序、即时消息收发应用程序、锻炼支持应用程序、照片管理应用程序、数码相机应用程序、数字摄影机应用程序、web浏览应用程序、数字音乐播放器应用程序和/或数字视频播放器应用程序。
可以在电子设备上执行的各种应用程序可以使用诸如触摸敏感表面的至少一个公共物理用户接口设备。可以在应用程序之间和/或相应应用程序内调整和/或改变触摸敏感表面的一个或多个功能以及电子设备上显示的相应信息。这样，电子设备的公共物理架构(例如，触摸敏感表面)可以支持具有对用户而言直观且透明的用户界面的各种应用程序。
应理解，本实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
为了说明本申请所述的技术方案，下面通过具体实施例来进行说明。
参见图1，图1是本申请实施例提供的一种基于数据库集群的数据处理方法的流程图一。该基于数据库集群的数据处理方法具体通过一实施监控组件来实现，或者借助已设计的其他硬件或软件来实现。
如图1所示，一种基于数据库集群的数据处理方法，该方法包括以下步骤：
步骤101，获取Redis数据库集群中主数据库的操作日志，依次将该操作日志分发至Redis数据库集群中的各从数据库。
Redis数据库集群结构中，数据库分为两类：主数据库(master)和从数据库(slave)。一个master可以拥有多个slave，但是一个slave只能对应一个master。
在该步骤中，对从数据库的数据进行同步，具体是通过实时监控组件从主数据库处获取到的操作日志来实现。通过实时监控组件，实现将操作日志在主数据库与从数据库之间进行传递，以使从数据库能够基于主数据库的操作日志，实现自身的数据同步更新。
该操作日志中记录有主数据库自身的数据更新操作，从数据在获取到分发的操作日志时，可以基于操作日志中的数据更新操作进行本地数据更新，该数据更新操作包括但不限于数据删除、数据新建、数据修改、数据间关系的建立、数据写操作、数据表的创建、数据表的导入等。
其中，实时监控组件在获取到操作日志之后，可以将其保存在本地，具体可以是将保存在本地的操作日志保留设定时长后进行删除，或者保留的操作日志的数量维持在一个设定值，超出设定值时则将先获取的操作日志删除。
具体地，该获取Redis数据库集群中主数据库的操作日志，具体为按照预设频率进行获取，或者当检测到主数据库中有新的操作日志产生时进行获取，或者当检测到主数据库中发生数据更新操作时，每当监控到这些数据操作，则需要对主数据库中的操作日志获取一次。
其中，作为一可选的实施方式，该获取Redis数据库集群中主数据库的操作日志，依次将操作日志分发至Redis数据库集群中的各从数据库，包括：
获取Redis数据库集群中主数据库的操作日志；对操作日志按照获取顺序进行序号标记，得到日志编号；依次将包含日志编号的操作日志分发至Redis数据库集群中的各从数据库。
其中日志编号为按照操作日志获取的先后顺序进行排序的编号。日志编号与操作日志的获取顺序一一对应。
实时监控组件在对主数据库的操作日志进行获取时，需要对得到的操作日志进行序号标记，标记的序号用于标识操作日志的获取次序，并在将操作日志向从数据库分发时依照操作日志的获取次序依次进行分发。
具体地，这里需要主数据库开启日志功能，实时监控组件获取的每条主数据库的操作日志都会生成一个唯一的带顺序的编号topicid。从数据库也需要开启日志功能。实时监控组件上配置好主库和从库的ip，实现在主库与从库之间进行信息传递，以实时抽取主数据库日志分发给从数据库。
步骤102，在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取第一从数据库发送的第一操作日志同步请求。
第一从数据库在接收到当前最新被分发的第一操作日志时，还未对该第一操作日志进行本地存储，需要将该最新被分发的第一操作日志与上一被分发的第二操作日志之间的关系进行判断，判断两者是否为连续的操作日志。其中，第一从数据库在确定两者为连续日志时，对该第一操作日志进行本地存储，并执行该第一操作日志，实现与主数据库之间的数据一致性。
当判断第一从数据库当前最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，则可以确定相邻两个操作日志之间存在日志缺失。
其中，对是否为连续日志的判断，从数据库可以通过对实时监控组件分发来的操作日志的日志编号进行比对，判断日志编号是否为连续编号，来确定操作日志是否为连续日志。判断本地存储的第二操作日志的日志编号与第一操作日志的日志编号是否为连续的编号，若是，则确定两个操作日志为连续日志，反之亦然。
例如：如果从数据库的topicid是t-001，实时监控组件推送过来的第一操作日志的topicid是t-003，则可得知t-003是晚于t-001生成的，且可以判断中间缺失了topicid为t-002的操作日志，则可以该当前接收的第一操作日志与本地存储的最新操作日志不为连续的日志。
产生操作日志不连续的原因，有可能是网络中断造成数据缺失，或原主库修复重启变成从数据库后，本身记录的topicid已不是最新的并由此造成数据缺失。
具体地，作为一可选的实施方式，该第一操作日志同步请求中携带有第一操作日志的第一日志编号及第二操作日志的第二日志编号；其中，第一日志编号与第二日志编号不连续。
即，从数据库在向实时监控组件发送操作日志同步请求时，可以将操作日志的顺序标号随同分发。
步骤103，根据第一操作日志同步请求，获取分发次序位于第一操作日志与第二操作日志之间的第一目标操作日志。
同样地，该分发次序位于第一操作日志与第二操作日志之间的第一目标操作日志的日志编号，为介于第一操作日志的日志编号与第二操作日志的日志编号之间。
即，该根据第一操作日志同步请求，获取分发次序位于第一操作日志与第二操作日志之间的第一目标操作日志，包括：
根据第一操作日志同步请求，获取日志编号介于第一日志编号及第二日志编号之间的第一目标操作日志。
在判断存在操作日志缺失的情况下，实时监控组件获取分发次序位于第一操作日志与第二操作日志之间的第一目标操作日志，即日志编号位于第一操作日志与第二操作日志的日志编号之间的目标操作日志，分发至第一从数据库，第一从数据库在接收到该些目标操作日志之后，则对该些日志数据进行解析并顺序执行。
比如从数据库当前本地存储的最新操作日志的topicid是t-003，实时监控组件中最后下发的操作日志的topicid已经是t-005，从数据库调用实时监控组件中的fresh()接口，入参是t-003，实时监控组件就会循环取出日志记录表中t-004和t-005日志返回给发出指令的从数据库，从数据库按顺序解析日志数据进行同步操作，记录最后一条操作日志的topicid，以保证从数据库存储的数据与主数据库的数据一致。
从数据库在每次执行日志同步操作后也会更新记录该最新的操作日志的topicid，用于后续基于该日志编号，保证自己和主库的数据是一致的。
步骤104，将第一目标操作日志分发至第一从数据库。
实现找出缺失的操作日志，并将其分发至从数据库中，确保从数据库中数据与主数据库中数据的一致性。
本申请实施例中，通过获取Redis数据库集群中主数据库的操作日志，依次将操作日志分发至Redis数据库集群中的各从数据库，在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取第一从数据库发送的第一操作日志同步请求，根据该第一操作日志同步请求，获取分发次序位于第一操作日志与该第二操作日志之间的第一目标操作日志，将第一目标操作日志分发至第一从数据库。该过程通过引入实时监控组件实现对主数据库的实时监控，在从数据库中最新被分发的操作日志与本地存储的最新操作日志不连续时，依照操作日志的分发次序找出缺失的操作日志，并将其分发至从数据库中，确保从数据库中数据与主数据库中数据的一致性。
本申请实施例中，通过获取redis数据库集群中主数据库的操作日志，依次将操作日志分发至redis数据库集群中的各从数据库，在第一从数据库最新被分发的第一操作日志与本地存储的最新操作日志不连续时，获取第一从数据库发送的第一操作日志同步请求，根据该第一操作日志同步请求，获取分发次序位于第一操作日志与该最新操作日志之间的第一目标操作日志，将第一目标操作日志分发至第一从数据库。该过程通过引入实时监控组件实现对主数据库的实时监控，在从数据库中最新被分发的操作日志与本地存储的最新操作日志不连续时，依照操作日志的分发次序找出缺失的操作日志，并将其分发至从数据库中，确保从数据库中数据与主数据库中数据的一致性。
本申请实施例中还提供了基于数据库集群的数据处理方法的不同实施方式。
参见图2，图2是本申请实施例提供的一种基于数据库集群的数据处理方法的流程图二。如图2所示，一种基于数据库集群的数据处理方法，该方法包括以下步骤：
步骤201，获取Redis数据库集群中主数据库的操作日志，依次将操作日志分发至Redis数据库集群中的各从数据库。
该步骤的实现过程与前述实施方式中的步骤101的实现过程相同，此处不再赘述。
步骤202，在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求。
该步骤的实现过程与前述实施方式中的步骤102的实现过程相同，此处不再赘述。
步骤203，根据该第一操作日志同步请求，获取分发次序位于第一操作日志与第二操作日志之间的第一目标操作日志。
该步骤的实现过程与前述实施方式中的步骤103的实现过程相同，此处不再赘述。
步骤204，将第一目标操作日志分发至第一从数据库。
该步骤的实现过程与前述实施方式中的步骤104的实现过程相同，此处不再赘述。
步骤205，在监控到主数据库出现故障时，确定最后向各个从数据库中分发的第三操作日志，及依次对各个从数据库中最后被分发的第四操作日志进行访问。
步骤206，根据第三操作日志及第四操作日志，从从数据库中确定第四操作日志与第三操作日志相一致的目标从数据库。
在监控到主数据库出现故障时，依次循环访问Redis数据库集群中的各从数据库，判断各从数据库中记录的最新的操作日志（即最后被分发的第四操作日志）与实时监控组件中的最新操作日志（即最后向各个从数据库中分发的第三操作日志）是否一致，若一致，则将当前从数据库设置为新的主数据库。
步骤207，将该主数据库识别为历史主数据库，将目标从数据库识别为新的主数据库。
该过程中，实时监控组件实时监控主数据库组件是否工作正常。实时监控组件实时监控到主数据库宕机时，立即从配置文件中循环ping所有配置的数据库ip，并比对从库的topicid和自己的topicid是否一致，如果topicid一致，则设当前从库为新的主库。
若不一致，则循环比对下一个从库ip。不一致的原因，可能是原来有问题的从库恢复使用了，或是由于网络故障从库没有接受到最新的操作日志。
本方案解决了由于主、从数据库同步过程中存在时间差导致数据不一致的问题，并可以迅速切换主从数据库从而大大缩短Redis数据库异常恢复时间，让用户使用时几乎无感，大大提高了Redis数据库集群的可用性。
在该步骤207执行完毕，则返回执行获取Redis数据库集群中主数据库的操作日志，依次将操作日志分发至Redis数据库集群中的各从数据库的步骤。
重新通过实施监控组件对新设置的主数据库进行监控，及协助进行主数据库与各从数据库之间的数据同步处理过程。
进一步地，在将目标从数据库识别为新的主数据库之后，还包括：
在检测到历史主数据库的故障被修复后，将历史主数据库识别为从数据库。
对应地，在将历史主数据库识别为从数据库之后，还包括：获取历史主数据库发送的第二操作日志同步请求；根据该第二操作日志同步请求，确定在历史主数据库出现故障之前从历史主数据库中最后获取的第五操作日志；根据该第五操作日志，获取分发次序位于第五操作日志与当前最新分发至各个从数据库的操作日志之间的第二目标操作日志；将该第二目标操作日志分发至历史主数据库。
实时监控组件监控到历史主数据库组件恢复后，自动把其识别为从数据库，并分发新的主数据库的日志给到历史主数据库进行同步操作。
具体地，实时监控组件获取历史主数据库最后执行的操作日志与实时监控组件从新的主数据库处获取的最新操作日志之间产生的操作日志，将其分发给已被识别为从数据库的历史主数据库，历史主数据库解析该些操作日志，并执行对应的操作，实现历史主数据库与新的主数据库之间的数据一致性。
历史数据库组件修复重启后，调用实时监控组件fresh()接口获取自己的最后执行的topicid和实时监控组件的最新topicid之间的日志数据进行解析并顺序执行，以保证自己的数据与新的主库数据保持一致。
该过程，可以大大缩短Redis数据库异常恢复时间，让用户使用时几乎无感，大大提高了Redis的可用性。
本申请实施例中，通过获取Redis数据库集群中主数据库的操作日志，依次将操作日志分发至Redis数据库集群中的各从数据库，在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取第一从数据库发送的第一操作日志同步请求，根据该第一操作日志同步请求，获取分发次序位于第一操作日志与该第二操作日志之间的第一目标操作日志，将第一目标操作日志分发至第一从数据库，在监控到主数据库出现故障时，将主数据库识别为历史主数据库，将目标从数据库识别为新的主数据库。该过程通过引入实时监控组件实现对主数据库的实时监控，在从数据库中最新被分发的操作日志与本地存储的最新操作日志不连续时，依照操作日志的分发次序找出缺失的操作日志，并将其分发至从数据库中，确保从数据库中数据与主数据库中数据的一致性，解决了由于主、从数据库同步过程中存在时间差导致数据不一致的问题，并可以迅速切换主从数据库从而大大缩短Redis数据库异常恢复时间，让用户使用时几乎无感，大大提高了Redis的可用性。
参见图3，图3是本申请实施例提供的一种基于数据库集群的数据处理装置的结构图，为了便于说明，仅示出了与本申请实施例相关的部分。
该基于数据库集群的数据处理装置300包括：
第一获取模块301，用于获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库；
第二获取模块302，用于在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求；
第三获取模块303，用于根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述第二操作日志之间的第一目标操作日志；
分发模块304，用于将所述第一目标操作日志分发至所述第一从数据库。
其中，第一获取模块，具体用于：
获取Redis数据库集群中主数据库的操作日志；
对所述操作日志按照获取顺序进行序号标记，得到日志编号；
依次将包含所述日志编号的所述操作日志分发至所述Redis数据库集群中的各从数据库。
其中，所述第一操作日志同步请求中携带有所述第一操作日志的第一日志编号及所述第二操作日志的第二日志编号；
其中，所述第一日志编号与所述第二日志编号不连续。
其中，第三获取模块303，具体用于：
根据所述第一操作日志同步请求，获取日志编号介于所述第一日志编号及所述第二日志编号之间的第一目标操作日志。
该装置还包括：数据库识别模块，用于：
在监控到所述主数据库出现故障时，确定最后向各个所述从数据库中分发的第三操作日志，及依次对各个所述从数据库中最后被分发的第四操作日志进行访问；
根据所述第三操作日志及所述第四操作日志，从所述从数据库中确定所述第四操作日志与所述第三操作日志相一致的目标从数据库；
将所述主数据库识别为历史主数据库，将所述目标从数据库识别为新的主数据库；
返回执行所述获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库的步骤。
其中，数据库识别模块，还用于：
在检测到所述历史主数据库的故障被修复后，将所述历史主数据库识别为从数据库。
该装置还包括：数据同步模块，用于：
获取所述历史主数据库发送的第二操作日志同步请求；
根据所述第二操作日志同步请求，确定在所述历史主数据库出现故障之前从所述历史主数据库中最后获取的第五操作日志；
根据所述第五操作日志，获取分发次序位于所述第五操作日志与当前最新分发至各个所述从数据库的操作日志之间的第二目标操作日志；
将所述第二目标操作日志分发至所述历史主数据库。
本申请实施例提供的基于数据库集群的数据处理装置能够实现上述基于数据库集群的数据处理方法的实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
图4是本申请实施例提供的一种电子设备的结构图。如该图所示，该实施例的电子设备4包括：至少一个处理器40（图4中仅示出一个）、存储器41以及存储在所述存储器41中并可在所述至少一个处理器40上运行的计算机程序42，所述处理器40执行所述计算机程序42时实现上述任意各个方法实施例中的步骤。
所述电子设备4可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。所述电子设备4可包括，但不仅限于，处理器40、存储器41。本领域技术人员可以理解，图4仅仅是电子设备4的示例，并不构成对电子设备4的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件，例如所述电子设备还可以包括输入输出设备、网络接入设备、总线等。
所述处理器40可以是中央处理单元(Central
Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable
Gate Array，FPGA) 或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
所述存储器41可以是所述电子设备4的内部存储单元，例如电子设备4的硬盘或内存。所述存储器41也可以是所述电子设备4的外部存储设备，例如所述电子设备4上配备的插接式硬盘，智能存储卡（Smart
Media Card，SMC），安全数字（Secure
Digital，SD）卡，闪存卡（Flash
Card）等。进一步地，所述存储器41还可以既包括所述电子设备4的内部存储单元也包括外部存储设备。所述存储器41用于存储所述计算机程序以及所述电子设备所需的其他程序和数据。所述存储器41还可以用于暂时地存储已经输出或者将要输出的数据。
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，仅以上述各功能单元、模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能单元、模块完成，即将所述装置的内部结构划分成不同的功能单元或模块，以完成以上描述的全部或者部分功能。实施例中的各功能单元、模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中，上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。另外，各功能单元、模块的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。上述系统中单元、模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述或记载的部分，可以参见其它实施例的相关描述。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
在本申请所提供的实施例中，应该理解到，所揭露的装置/电子设备和方法，可以通过其它的方式实现。例如，以上所描述的装置/电子设备实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通讯连接可以是通过一些接口，装置或单元的间接耦合或通讯连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的模块/单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实现上述实施例方法中的全部或部分流程，也可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质可以包括：能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器（ROM，Read-Only Memory）、随机存取存储器（RAM，Random Access Memory）、电载波信号、电信信号以及软件分发介质等。需要说明的是，所述计算机可读介质包含的内容可以根据司法管辖区内立法和专利实践的要求进行适当的增减，例如在某些司法管辖区，根据立法和专利实践，计算机可读介质不包括电载波信号和电信信号。所述计算机可读存储介质可以是非易失性，也可以是易失性。
本申请实现上述实施例方法中的全部或部分流程，也可以通过计算机程序产品来实现，当计算机程序产品在电子设备上运行时，使得所述电子设备执行时实现可实现上述各个方法实施例中的步骤。
以上所述实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围，均应包含在本申请的保护范围之内。
Claims (20)
- 一种基于数据库集群的数据处理方法，其中，包括：获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库；在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求；根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述第二操作日志之间的第一目标操作日志；将所述第一目标操作日志分发至所述第一从数据库。
- 根据权利要求1所述的数据处理方法，其中，所述获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库，包括：获取Redis数据库集群中主数据库的操作日志；对所述操作日志按照获取顺序进行序号标记，得到日志编号；依次将包含所述日志编号的所述操作日志分发至所述Redis数据库集群中的各从数据库。
- 根据权利要求2所述的数据处理方法，其中，所述第一操作日志同步请求中携带有所述第一操作日志的第一日志编号及所述第二操作日志的第二日志编号；其中，所述第一日志编号与所述第二日志编号不连续。
- 根据权利要求3所述的数据处理方法，其中，所述根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述第二操作日志之间的第一目标操作日志，包括：根据所述第一操作日志同步请求，获取日志编号介于所述第一日志编号及所述第二日志编号之间的第一目标操作日志。
- 根据权利要求1所述的数据处理方法，其中，所述将所述第一目标操作日志分发至所述第一从数据库之后，还包括：在监控到所述主数据库出现故障时，确定最后向各个所述从数据库中分发的第三操作日志，及依次对各个所述从数据库中最后被分发的第四操作日志进行访问；根据所述第三操作日志及所述第四操作日志，从所述从数据库中确定所述第四操作日志与所述第三操作日志相一致的目标从数据库；将所述主数据库识别为历史主数据库，将所述目标从数据库识别为新的主数据库；返回执行所述获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库的步骤。
- 根据权利要求5所述的数据处理方法，其中，所述将所述目标从数据库识别为新的主数据库之后，还包括：在检测到所述历史主数据库的故障被修复后，将所述历史主数据库识别为从数据库。
- 根据权利要求6所述的数据处理方法，其中，所述将所述历史主数据库识别为从数据库之后，还包括：获取所述历史主数据库发送的第二操作日志同步请求；根据所述第二操作日志同步请求，确定在所述历史主数据库出现故障之前从所述历史主数据库中最后获取的第五操作日志；根据所述第五操作日志，获取分发次序位于所述第五操作日志与当前最新分发至各个所述从数据库的操作日志之间的第二目标操作日志；将所述第二目标操作日志分发至所述历史主数据库。
- 一种基于数据库集群的数据处理装置，其中，包括：第一获取模块，用于获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库；第二获取模块，用于在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求；第三获取模块，用于根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述第二操作日志之间的第一目标操作日志；分发模块，用于将所述第一目标操作日志分发至所述第一从数据库。
- 根据权利要求8所述的数据处理装置，其中，所述第一获取模块，具体用于：获取Redis数据库集群中主数据库的操作日志；对所述操作日志按照获取顺序进行序号标记，得到日志编号；依次将包含所述日志编号的所述操作日志分发至所述Redis数据库集群中的各从数据库。
- 根据权利要求9所述的数据处理装置，其中，所述第一操作日志同步请求中携带有所述第一操作日志的第一日志编号及所述第二操作日志的第二日志编号；其中，所述第一日志编号与所述第二日志编号不连续。
- 根据权利要求10所述的数据处理装置，其中，所述第三获取模块，具体用于：根据所述第一操作日志同步请求，获取日志编号介于所述第一日志编号及所述第二日志编号之间的第一目标操作日志。
- 根据权利要求8所述的数据处理装置，其中，所述装置还包括：数据库识别模块，用于：在监控到所述主数据库出现故障时，确定最后向各个所述从数据库中分发的第三操作日志，及依次对各个所述从数据库中最后被分发的第四操作日志进行访问；根据所述第三操作日志及所述第四操作日志，从所述从数据库中确定所述第四操作日志与所述第三操作日志相一致的目标从数据库；将所述主数据库识别为历史主数据库，将所述目标从数据库识别为新的主数据库；返回执行所述获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库的步骤。
- 一种电子设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其中，所述处理器执行所述计算机程序时实现如下步骤：获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库；在第一从数据库最新被分发的第一操作日志与上一被分发的第二操作日志不连续时，获取所述第一从数据库发送的第一操作日志同步请求；根据所述第一操作日志同步请求，获取分发次序位于所述第一操作日志与所述第二操作日志之间的第一目标操作日志；将所述第一目标操作日志分发至所述第一从数据库。
- 根据权利要求13所述的电子设备，所述处理器执行所述计算机程序时具体实现如下步骤：获取Redis数据库集群中主数据库的操作日志；对所述操作日志按照获取顺序进行序号标记，得到日志编号；依次将包含所述日志编号的所述操作日志分发至所述Redis数据库集群中的各从数据库。
- 根据权利要求14所述的电子设备，其中，所述第一操作日志同步请求中携带有所述第一操作日志的第一日志编号及所述第二操作日志的第二日志编号；其中，所述第一日志编号与所述第二日志编号不连续。
- 根据权利要求15所述的电子设备，所述处理器执行所述计算机程序时具体实现如下步骤：根据所述第一操作日志同步请求，获取日志编号介于所述第一日志编号及所述第二日志编号之间的第一目标操作日志。
- 根据权利要求13所述的电子设备，所述处理器执行所述计算机程序时还实现如下步骤：在监控到所述主数据库出现故障时，确定最后向各个所述从数据库中分发的第三操作日志，及依次对各个所述从数据库中最后被分发的第四操作日志进行访问；根据所述第三操作日志及所述第四操作日志，从所述从数据库中确定所述第四操作日志与所述第三操作日志相一致的目标从数据库；将所述主数据库识别为历史主数据库，将所述目标从数据库识别为新的主数据库；返回执行所述获取Redis数据库集群中主数据库的操作日志，依次将所述操作日志分发至所述Redis数据库集群中的各从数据库的步骤。
- 根据权利要求17所述的电子设备，所述处理器执行所述计算机程序时还实现如下步骤：在检测到所述历史主数据库的故障被修复后，将所述历史主数据库识别为从数据库。
- 根据权利要求18所述的电子设备，所述处理器执行所述计算机程序时还实现如下步骤：获取所述历史主数据库发送的第二操作日志同步请求；根据所述第二操作日志同步请求，确定在所述历史主数据库出现故障之前从所述历史主数据库中最后获取的第五操作日志；根据所述第五操作日志，获取分发次序位于所述第五操作日志与当前最新分发至各个所述从数据库的操作日志之间的第二目标操作日志；将所述第二目标操作日志分发至所述历史主数据库。
- 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其中，所述计算机程序被处理器执行时实现如权利要求1至7任一项所述方法的步骤。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011588369.0A CN112612853A (zh) | 2020-12-28 | 2020-12-28 | 基于数据库集群的数据处理方法、装置及电子设备 |
| CN202011588369.0 | 2020-12-28 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022142665A1 true WO2022142665A1 (zh) | 2022-07-07 |
Family
ID=75248592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2021/127181 WO2022142665A1 (zh) | 2020-12-28 | 2021-10-28 | 基于数据库集群的数据处理方法、装置及电子设备 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112612853A (zh) |
| WO (1) | WO2022142665A1 (zh) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115438087A (zh) * | 2022-11-10 | 2022-12-06 | 广州思迈特软件有限公司 | 基于缓存库的数据查询方法、装置、存储介质和设备 |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112612853A (zh) * | 2020-12-28 | 2021-04-06 | 深圳壹账通智能科技有限公司 | 基于数据库集群的数据处理方法、装置及电子设备 |
| CN113177093A (zh) * | 2021-05-24 | 2021-07-27 | 商汤国际私人有限公司 | 数据同步方法及装置、电子设备和存储介质 |
| CN113806301A (zh) * | 2021-09-29 | 2021-12-17 | 中国平安人寿保险股份有限公司 | 数据同步方法、装置、服务器及存储介质 |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040267809A1 (en) * | 2003-06-23 | 2004-12-30 | Microsoft Corporation | Resynchronization of multiple copies of a database after a divergence in transaction history |
| CN101751394A (zh) * | 2008-12-16 | 2010-06-23 | 青岛海信传媒网络技术有限公司 | 数据同步方法和数据同步系统 |
| CN108984569A (zh) * | 2017-06-05 | 2018-12-11 | 中兴通讯股份有限公司 | 数据库切换方法、系统和计算机可读存储介质 |
| CN110019062A (zh) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | 数据同步方法和系统 |
| CN112612853A (zh) * | 2020-12-28 | 2021-04-06 | 深圳壹账通智能科技有限公司 | 基于数据库集群的数据处理方法、装置及电子设备 |
-
2020
- 2020-12-28 CN CN202011588369.0A patent/CN112612853A/zh active Pending
-
2021
- 2021-10-28 WO PCT/CN2021/127181 patent/WO2022142665A1/zh active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040267809A1 (en) * | 2003-06-23 | 2004-12-30 | Microsoft Corporation | Resynchronization of multiple copies of a database after a divergence in transaction history |
| CN101751394A (zh) * | 2008-12-16 | 2010-06-23 | 青岛海信传媒网络技术有限公司 | 数据同步方法和数据同步系统 |
| CN108984569A (zh) * | 2017-06-05 | 2018-12-11 | 中兴通讯股份有限公司 | 数据库切换方法、系统和计算机可读存储介质 |
| CN110019062A (zh) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | 数据同步方法和系统 |
| CN112612853A (zh) * | 2020-12-28 | 2021-04-06 | 深圳壹账通智能科技有限公司 | 基于数据库集群的数据处理方法、装置及电子设备 |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115438087A (zh) * | 2022-11-10 | 2022-12-06 | 广州思迈特软件有限公司 | 基于缓存库的数据查询方法、装置、存储介质和设备 |
| CN115438087B (zh) * | 2022-11-10 | 2023-03-24 | 广州思迈特软件有限公司 | 基于缓存库的数据查询方法、装置、存储介质和设备 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112612853A (zh) | 2021-04-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022142665A1 (zh) | 基于数据库集群的数据处理方法、装置及电子设备 | |
| JP6602369B2 (ja) | 記憶不具合後の安全なデータアクセス | |
| US20100199065A1 (en) | Methods and apparatus for performing efficient data deduplication by metadata grouping | |
| CN110096302B (zh) | 软件数据储存、升级方法、装置、设备和存储介质 | |
| CN103597440A (zh) | 用于创建克隆文件的方法以及采用该方法的文件系统 | |
| CN102272751B (zh) | 在数据库环境通过背景同步的数据完整性 | |
| WO2022095366A1 (zh) | 基于Redis的数据读取方法、装置、设备及可读存储介质 | |
| CN110737682A (zh) | 一种缓存操作方法、装置、存储介质和电子设备 | |
| WO2020133995A1 (zh) | 副本部署方法、云服务器及存储介质 | |
| CN108717516B (zh) | 文件标签方法、终端及介质 | |
| WO2024041022A1 (zh) | 数据库表变更方法、装置、设备和存储介质 | |
| CN110633046A (zh) | 一种分布式系统的存储方法、装置、存储设备及存储介质 | |
| CN112256676A (zh) | 一种数据库迁移的方法、装置、设备和介质 | |
| CN111767270A (zh) | 数据迁移方法、装置、服务器及存储介质 | |
| WO2018010603A1 (zh) | 基于视频云存储系统的存储模式升级方法、装置和系统 | |
| CN111475334A (zh) | TiDB数据库的维护方法、装置、计算机设备及存储介质 | |
| CN112181942A (zh) | 时序数据库系统和数据处理方法及装置 | |
| CN116208487A (zh) | 区块链系统中的共识算法升级方法、装置、设备及介质 | |
| CN115640280A (zh) | 数据迁移方法及装置 | |
| CN115964436A (zh) | 一种数据库集群管理的方法和装置 | |
| US11341159B2 (en) | In-stream data load in a replication environment | |
| CN113656378A (zh) | 一种服务器管理方法、装置、介质 | |
| CN111694801A (zh) | 一种应用于故障恢复的数据去重方法和装置 | |
| CN112787849B (zh) | 区块链状态的管控方法、装置、终端及存储介质 | |
| CN113868679B (zh) | 一种集群的加密方法及装置 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21913409 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 021023) |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 21913409 Country of ref document: EP Kind code of ref document: A1 |