CN116841983A - Data rollback method, device and computer readable storage medium - Google Patents
Data rollback method, device and computer readable storage medium Download PDFInfo
- Publication number
- CN116841983A CN116841983A CN202310783735.5A CN202310783735A CN116841983A CN 116841983 A CN116841983 A CN 116841983A CN 202310783735 A CN202310783735 A CN 202310783735A CN 116841983 A CN116841983 A CN 116841983A
- Authority
- CN
- China
- Prior art keywords
- data
- rollback
- log
- change information
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 230000008859 change Effects 0.000 claims abstract description 203
- 238000012544 monitoring process Methods 0.000 claims abstract description 87
- 239000000523 sample Substances 0.000 claims description 45
- 230000008569 process Effects 0.000 claims description 20
- 230000004083 survival effect Effects 0.000 claims description 11
- 238000005096 rolling process Methods 0.000 claims description 9
- 238000004519 manufacturing process Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/526—Mutual exclusion algorithms
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data rollback method, a device and a computer readable storage medium, wherein the method comprises the following steps: monitoring a binary log of a database, and determining data change information corresponding to change data in the database according to the binary log; when a rollback instruction is received, determining target data change information corresponding to the rollback instruction; and reversely generating a rollback code according to the target data change information, and executing the rollback code. The invention aims to improve the data rollback efficiency.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a data rollback method, apparatus, and computer readable storage medium.
Background
In database operation, after data modification, the change condition of the data needs to be recorded in time, and when the conditions such as data error change and false deletion occur, the data rollback needs to be performed. In the current data rollback scheme, data is directly rolled back through a database, and data content, designated time, version and the like needing to be rolled back need to be defined in the mode, so that a great deal of time is required for rollback, and the data rollback efficiency is low.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide a data rollback method, a data rollback device and a computer readable storage medium, which aim to achieve the effect of improving the data rollback efficiency.
In order to achieve the above object, the present invention provides a data rollback method, including:
monitoring a binary log of a database, and determining data change information corresponding to change data in the database according to the binary log;
when a rollback instruction is received, determining target data change information corresponding to the rollback instruction;
and reversely generating a rollback code according to the target data change information, and executing the rollback code.
Optionally, the step of monitoring a binary log of the database and determining, according to the binary log, data change information corresponding to change data in the database includes:
deploying log monitoring application based on the application instance of the container management platform;
triggering a ready probe to fight against the distributed lock when the application instance is ready;
when the race is successful, initializing a configuration item of the log monitoring application based on the application instance, and monitoring the binary log based on the configuration item;
and triggering a survival probe in the running process of the application instance, and renewing the distributed lock so that the log monitoring application continuously monitors the binary log.
Optionally, after the step of triggering a survival probe during the running process of the application instance and renewing the distributed lock to enable the log listening application to continually listen to the binary log, the method further includes:
triggering a stop probe when the stop condition of monitoring the binary log is met, determining the current monitored log position of the log monitoring application based on the stop probe, and stopping the current application instance;
when the next application instance is ready, taking the log position as a monitoring starting position corresponding to a log monitoring application in the next application instance;
wherein the stop condition of the listening archive log comprises at least one of:
the application instance fails, restarts, redeploys;
and receiving a stop instruction for monitoring the archive log.
Optionally, the step of initializing a configuration item of the log listening application based on the application instance includes:
acquiring a database object set, fields, field types and positions of fields in a table of each data table in the database;
and taking the database object set, the fields, the field types and the positions of the fields in a data table as the configuration items.
Optionally, after the step of monitoring the binary log of the database and determining the data change information corresponding to the change data in the database according to the binary log, the method further includes:
sending the data change information to a consumption queue as a consumption event;
polling data change information in the consumption queue based on a plurality of consumption threads, and binding a batch number identifier for the data change information;
and storing the data change information after the batch number identification is bound in a memory.
Optionally, before the step of polling the data change information in the consumption queue based on the plurality of consumption threads and binding the lot number identification for the data change information, the method further includes:
acquiring a transaction identifier corresponding to a current transaction of the database;
and carrying out association binding on the batch number identification corresponding to the change data in the current transaction and the transaction identification.
Optionally, the step of determining the target change data corresponding to the rollback instruction includes:
determining a target transaction identifier according to the rollback instruction, determining a batch number identifier which is associated and bound with the target transaction identifier, and taking data change information corresponding to the batch number identifier as the target data change information; or alternatively
And determining a batch number identification according to the rollback instruction, and taking data change information corresponding to the batch number identification as the target data change information.
Optionally, after the step of monitoring the binary log of the database and determining the data change information corresponding to the change data in the database according to the binary log, the method further includes:
determining information to be displayed corresponding to each change data according to the data change information;
generating a graphical interface according to the information to be displayed;
and receiving a rollback instruction of a user based on the graphical interface.
In addition, in order to achieve the above object, the present invention also provides a data rollback apparatus, which includes a memory, a processor, and a data rollback program stored on the memory and executable on the processor, wherein the data rollback program implements the steps of the data rollback method as described above when executed by the processor.
In addition, to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a data rollback program which, when executed by a processor, implements the steps of the data rollback method as described above.
According to the method, binary logs of a database are monitored, and data change information corresponding to change data in the database is determined according to the binary logs; when a rollback instruction is received, determining target data change information corresponding to the rollback instruction; and reversely generating a rollback code according to the target data change information, and executing the rollback code. The binary log of the database is monitored to obtain the data change information corresponding to the change data, a user can designate target change information needing to be rolled back through a rolling-back instruction, automatically and reversely roll back codes according to the target change information, and the rolling-back codes are executed to realize rolling back of the change data corresponding to the target data change information, so that the data rolling-back efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of a terminal structure of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flow chart of an embodiment of a data rollback method according to the present invention;
FIG. 3 is a flow chart of another embodiment of the data rollback method of the present invention;
FIG. 4 is a schematic diagram of a first application field Jing Liucheng according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an initialization configuration item according to an embodiment of the present invention;
fig. 6 is a schematic flow chart of a second application scenario according to an embodiment of the present invention;
FIG. 7 is a diagram of a relationship between identification and data table according to an embodiment of the present invention;
FIG. 8 is a graphical interface according to an embodiment of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Because the existing data rollback schemes are all used for directly rolling back data through a database, the data content, the designated time, the version and the like which need to be rolled back need to be defined in the mode, and therefore a great deal of time is consumed for rollback, and the data rollback efficiency is low.
In order to improve data rollback efficiency, embodiments of the present invention provide a data rollback method, apparatus, and computer readable storage medium, where main steps of the method include:
monitoring a binary log of a database, and determining data change information corresponding to change data in the database according to the binary log;
when a rollback instruction is received, determining target data change information corresponding to the rollback instruction;
and reversely generating a rollback code according to the target data change information, and executing the rollback code.
The binary log of the database is monitored to obtain the data change information corresponding to the change data, a user can designate target change information needing to be rolled back through a rolling-back instruction, automatically and reversely roll back codes according to the target change information, and the rolling-back codes are executed to realize rolling back of the change data corresponding to the target data change information, so that the data rolling-back efficiency is improved.
The invention as claimed is described in detail below with reference to the attached drawing figures.
As shown in fig. 1, fig. 1 is a schematic diagram of a terminal structure of a hardware running environment according to an embodiment of the present invention.
The terminal of the embodiment of the invention can be a data rollback device.
As shown in fig. 1, the terminal may include: a processor 1001, such as a CPU, a memory 1003, and a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The memory 1003 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1003 may alternatively be a storage device separate from the processor 1001 described above.
It will be appreciated by those skilled in the art that the terminal structure shown in fig. 1 is not limiting of the terminal and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system and a data rollback program may be included in a memory 1003, which is a kind of computer storage medium.
In the terminal shown in fig. 1, the processor 1001 may be configured to call a data rollback program stored in the memory 1003, and perform the following operations:
monitoring a binary log of a database, and determining data change information corresponding to change data in the database according to the binary log;
when a rollback instruction is received, determining target data change information corresponding to the rollback instruction;
and reversely generating a rollback code according to the target data change information, and executing the rollback code.
Further, the processor 1001 may call a data rollback program stored in the memory 1003, and further perform the following operations:
deploying log monitoring application based on the application instance of the container management platform;
triggering a ready probe to fight against the distributed lock when the application instance is ready;
when the race is successful, initializing a configuration item of the log monitoring application based on the application instance, and monitoring the binary log based on the configuration item;
and triggering a survival probe in the running process of the application instance, and renewing the distributed lock so that the log monitoring application continuously monitors the binary log.
Further, the processor 1001 may call a data rollback program stored in the memory 1003, and further perform the following operations:
triggering a stop probe when the stop condition of monitoring the binary log is met, determining the current monitored log position of the log monitoring application based on the stop probe, and stopping the current application instance;
when the next application instance is ready, taking the log position as a monitoring starting position corresponding to a log monitoring application in the next application instance;
wherein the stop condition of the listening archive log comprises at least one of:
the application instance fails, restarts, redeploys;
and receiving a stop instruction for monitoring the archive log.
Further, the processor 1001 may call a data rollback program stored in the memory 1003, and further perform the following operations:
acquiring a database object set, fields, field types and positions of fields in a table of each data table in the database;
and taking the database object set, the fields, the field types and the positions of the fields in a data table as the configuration items.
Further, the processor 1001 may call a data rollback program stored in the memory 1003, and further perform the following operations:
sending the data change information to a consumption queue as a consumption event;
polling data change information in the consumption queue based on a plurality of consumption threads, and binding a batch number identifier for the data change information;
and storing the data change information after the batch number identification is bound in a memory.
Further, the processor 1001 may call a data rollback program stored in the memory 1003, and further perform the following operations:
acquiring a transaction identifier corresponding to a current transaction of the database;
and carrying out association binding on the batch number identification corresponding to the change data in the current transaction and the transaction identification.
Further, the processor 1001 may call a data rollback program stored in the memory 1003, and further perform the following operations:
determining a target transaction identifier according to the rollback instruction, determining a batch number identifier which is associated and bound with the target transaction identifier, and taking data change information corresponding to the batch number identifier as the target data change information; or alternatively
And determining a batch number identification according to the rollback instruction, and taking data change information corresponding to the batch number identification as the target data change information.
Further, the processor 1001 may call a data rollback program stored in the memory 1003, and further perform the following operations:
determining information to be displayed corresponding to each change data according to the data change information;
generating a graphical interface according to the information to be displayed;
and receiving a rollback instruction of a user based on the graphical interface.
The following is a description of what is claimed in the claims of the present invention by means of specific exemplary embodiments, so that those skilled in the art can better understand the scope of the claims of the present invention. It should be understood that the following exemplary embodiments do not limit the scope of the present invention, but are only used to illustrate the present invention.
Illustratively, referring to FIG. 2, in one embodiment of the data rollback method of the present invention, the data rollback method comprises the steps of:
s10, monitoring a binary log of a database, and determining data change information corresponding to change data in the database according to the binary log;
in this embodiment, the database is a collection of large amounts of data stored in a computer for a long period of time, organized, sharable, and uniformly managed. In the process of using the database, the situation of data error change, error deletion and the like is unavoidable, and the data change operation which is already performed needs to be withdrawn, so that the changed data is restored to the data before the change, namely, the data rollback. However, at present, the data rollback is directly rolled back through the database, and the following disadvantages exist:
1. the rollback action itself is time consuming, and the direct database rollback also defines what needs to be rolled back, what does not need to be rolled back, and the appointed time, version and the like are relatively very complicated;
2. there is no association with traffic data: the rollback is based on the database level, the accuracy experience of positioning data and checking data is poor, the affected data range cannot be checked through the view angle of a user, and a graphical interface cannot be provided for operation and maintenance means such as data screening, rollback operation and the like.
3. Full life cycle monitoring without database table data is not formed, and changed process data cannot be provided;
4. for the stock system, the data has no logical deletion field, and in the case of deleting the data by mistake, a faster data rollback operation cannot be provided.
The log monitoring application provides continuous database data change monitoring, the log monitoring application can be a binlog client of a mysql-binlog-connector open source component, a binlog log (binary log) of a database is obtained based on the binlog client to synchronize change data, analysis is carried out after the binlog is obtained, data change information of each line of change data of each table is analyzed, the data change information of each line of data of each table is stored, a storage layer can be selected according to actual conditions, and storage systems such as mysql, es, hbase can be selected.
Step S20, when a rollback instruction is received, determining target data change information corresponding to the rollback instruction;
in this embodiment, after the data change information is stored, when a user needs some data, for example, some line or some transaction data rollback, the data rollback needed to be performed in the data rollback instruction may be determined by triggering a rollback instruction corresponding to the data change, which may be the data change needed to be rolled back in some line or the data change needed to be rolled back corresponding to some transaction. And extracting the target data change information corresponding to the change data needing to be rolled back from the storage system.
And S30, reversely generating a rollback code according to the target data change information, and executing the rollback code.
In this embodiment, the data change information is detailed to the data before the data change and the data after the data change of the change data of each line in each specific table. The rollback code can be reversely generated based on the data change information, the rollback code refers to rollback SQL in a database, SQL (Structured Query Language) is a database language with multiple functions of data manipulation, data definition and the like, and the rollback code is executed, so that change data corresponding to a rollback instruction can be restored to a state before data change, and data rollback is realized.
In the technical scheme disclosed by the embodiment, by monitoring a binary log of a database, determining data change information corresponding to change data in the database according to the binary log; when a rollback instruction is received, determining target data change information corresponding to the rollback instruction; and reversely generating a rollback code according to the target data change information, and executing the rollback code. The binary log of the database is monitored to obtain the data change information corresponding to the change data, a user can designate target change information needing to be rolled back through a rolling-back instruction, automatically and reversely roll back codes according to the target change information, and the rolling-back codes are executed to realize rolling back of the change data corresponding to the target data change information, so that the data rolling-back efficiency is improved.
Further, step S10 includes:
deploying log monitoring application based on the application instance of the container management platform;
triggering a ready probe to fight against the distributed lock when the application instance is ready;
when the race is successful, initializing a configuration item of the log monitoring application based on the application instance, and monitoring the binary log based on the configuration item;
and triggering a survival probe in the running process of the application instance, and renewing the distributed lock so that the log monitoring application continuously monitors the binary log.
In this embodiment, in order to control the log listening application to listen to the binary log of the database, a container management platform, such as kubernetes, that can use a distributed deployment scenario of multiple AZ and multiple application instances of a service application may be used to manage the running of the log listening application, so as to improve the high availability of services. The log monitoring application is embedded into a normal service system, and is deployed based on an application instance (pod) of a kubernetes container management platform. Under the condition that N application instances survive, if a plurality of log monitoring applications start and collect logs at the same time, the same data change is liable to generate N pieces of data analyzed by binlog and stored in a storage system, and the problem of synchronous delay of a master database and a slave database of the database is also caused, so that the execution efficiency of the database is reduced, and the single-instance starting and execution of the log monitoring application are required.
The monitoring process of starting the log monitoring application to start monitoring and continuing the log monitoring application is realized by two probes: namely ready probes and survival probes. Referring to fig. 4, a log listening application is deployed based on an application instance of a container management platform; triggering a ready probe to fight against the distributed lock when the application instance is ready; when the race is successful, initializing a configuration item of a log monitoring application based on the application instance, monitoring a binary log based on the configuration item, triggering a survival probe in the running process of the application instance, and distributing a locking contract so that the log monitoring application continuously monitors the binary log.
Specifically, the kubernetes platform provides a readiness probe (ready probe) and a liveness probe (survival probe), deploys a binlog client (log snoop application) of a mysql-binlog-connector open source component based on a pod, triggers the readiness probe when the pod is successfully ready, competes for a distributed lock, initializes all relevant configuration items of the binlog client if the competition succeeds, the configuration item information is stored in the application memory, and the configuration items can comprise binlog client initialization, description identification of a database table, position and binlog file name acquired by the last log, and the like, and listens for the binlog based on the binlog client after initialization. Thus, the corresponding data change information is extracted from the binlog based on the configuration item, the log analysis data performance can be improved,
in the pod running process, namely in the running process of the application instance, the readinessProbe probe is successfully executed, and when the current pod competes for the distributed lock successfully, the distributed lock is contracted, so that only the current pod can normally acquire logs.
Therefore, the ready probe and the survival probe are used for competing against the distributed lock, so that the situation that a plurality of application instances repeatedly deploy log monitoring applications to collect the same binary log is avoided, the effects of single-instance operation and single-instance log collection are achieved, normal application of the monitoring log is ensured, and the data rollback efficiency is improved.
After the step of triggering the survival probe during the running process of the application instance and renewing the distributed lock to enable the log listening application to continually listen to the binary log, the method further comprises:
triggering a stop probe when the stop condition of monitoring the binary log is met, determining the current monitored log position of the log monitoring application based on the stop probe, and stopping the current application instance;
when the next application instance is ready, taking the log position as a monitoring starting position corresponding to a log monitoring application in the next application instance;
wherein the stop condition of the listening archive log comprises at least one of:
the application instance fails, restarts, redeploys;
and receiving a stop instruction for monitoring the archive log.
In this embodiment, when the log is deployed to monitor the application based on the application instance of the container management platform, the situation that the application instance needs to be stopped occurs, for example, when the application instance needs to be redeployed, restarted or hung, the current application instance needs to be paused, and a new application instance takes over the abnormal application instance and continues to collect the log. Thus, referring to fig. 4, when a stop condition of the snoop binary log is satisfied, a stop probe is triggered. The stop condition of the snoop binary log includes an application instance failure, a restart, a redeployment, and/or receipt of a stop instruction to the snoop binary log. When the stop probe is triggered, stopping the current application instance, determining the current monitored log position of the log monitoring application based on the stop probe, and taking the log position as the monitoring starting position corresponding to the log monitoring application in the next application instance when the next application instance is ready, and continuing monitoring of the database from the monitoring starting position to ensure the integrity of monitoring.
Specifically, the kubernetes platform further provides a preStop probe (stop probe) for stopping the operation of the pod, and when the pod is stopped, the log monitor application corresponding to the pod also stops running. When the pod stops, the preStop probe can be used for notifying the log monitoring application to record the currently acquired log position, when the next readiness probe is triggered, namely, when the next pod is ready, the log position is a configuration item for initializing the readiness probe of the next pod which operates normally, and the log is acquired from the log position.
For example, in the case where there is no change in the normal database table, there is no problem in normally collecting logs, but if the database table is modified when the DDL sql is executed by the DBA or database deployment, the binlog collection needs to track the field change of each table, so that each log collection can monitor the change of all fields, and no field data is lost during rollback. The process is matched with log monitoring, so that full automation is realized. This means that it is necessary to monitor the binary log of the database for a long time, and when the data in the database is changed frequently, the binlog will surge, and when the binlog file reaches a certain size, or after mysql is restarted, the database will regenerate a new binlog file record. This faces how too much binary log is handled and how the database continues to listen after switching binary log files:
for the problem of how to treat too much binary logs, a timing task can be set to realize timing monitoring, the timing task is executed once per hour, data in a historical period is cleaned, the generated log record data size is larger in view of the fact that the production environment is not shut down for operation and maintenance, the data in a more recent time period, such as data before 3 days, is required to be cleaned, the non-production environment data size is small, the cleaning time can be prolonged, such as cleaning 7 days of data, and the problem that the data is too much to store is prevented.
For the problem of how to continue monitoring after the database switches binary log files, when the current application instance runs successfully, the ready probe executes successfully, the binary log file name recorded by the current log monitoring application is initialized, the binary log name recorded currently is compared with the binary log file name acquired when the previous ready probe executes, if the file names are inconsistent, the database is indicated to switch the binary log files, then the log monitoring application monitors the binary log file corresponding to the newly determined binary log file name, and the starting position of the binary log file is used as the monitoring starting position corresponding to the log monitoring application in the application instance corresponding to the next ready probe. For example, when a timed task is executed, sql will be executed: show master status to obtain the current latest log binlog file, if the initialized binlog file name is inconsistent with the binlog file name of the latest log, then the binlog file of the mysql switch log is described, such as from binlog.000047 to binlog.000048. The binlog client listens for a new binlog file and records the binlog from scratch at the beginning of the log record, i.e., where position=4. And re-saving the binlog file name and position to the database for the next initialization of the binlog client.
Therefore, when the application instance fails, is restarted and is redeployed, and/or a stop instruction for monitoring the binary log is received, the monitoring of the database can be stopped through the stop probe, and the current monitored log position is recorded, and when the next application instance is ready, the monitoring is continued by the log position, so that the integrity of the monitored database is ensured, and the data rollback efficiency is improved.
Further, the step of initializing a configuration item of the log listening application based on the application instance includes:
acquiring a database object set, a field type and a position of a field in a table corresponding to each field in the database;
and taking the database object set, the fields, the field types and the positions of the fields in a data table as the configuration items.
In this embodiment, after triggering the ready probe, the configuration item for initializing the log monitoring application needs to be completed, and the binary log is monitored based on the configuration item, so that in order to ensure that the log monitoring process, after the binary log is analyzed after the acquisition arrives, the data change information needs to be saved through the database, and the data change information needs to be saved to the database to be consistent with the database of the acquired log. Specifically, referring to fig. 5, a database object set (schema), a field type, and a position of a field in a table of each data table in the collected database are acquired, and the database object set, the field type, and the position of the field in the table are used as configuration items of the initialized log monitoring application, wherein the field type also needs to contain an identifier of whether the field type is a primary key, because the deletion operation is performed based on the primary key.
Therefore, the consistency of the data change information and the fields of the database of the collected log can be ensured, the consistency of the fields of the rollback code and the fields of the database of the collected log is ensured, the change of the fields can be recorded in real time, and the initialization configuration items of the table description are stored in the local memory of the application, so that the acquisition is convenient, and the rollback efficiency of the data is improved.
Optionally, referring to fig. 3, in another embodiment of the data rollback method according to the present invention, after step S10, the data rollback method further includes:
step S40, the data change information is sent to a consumption queue as a consumption event;
step S50, polling data change information in the consumption queue based on a plurality of consumption threads, and binding a batch number identifier for the data change information;
step S60, the data change information marked by the binding batch number is stored in a memory.
In this embodiment, in order to ensure that the data change analysis can be performed most quickly, the module uses a production-consumer model to perform quick analysis, so that the log collection and analysis process is a computationally intensive task. The production end is a log monitoring application and is used for collecting the synchronous binary log of the database. After the production end analyzes the data to obtain data change information, the data change information is used as a consumption event and is transmitted to a local consumption queue. Multiple local consumption queues can be established, so that a batch number identifier is bound for each piece of data change information based on the data change information in the consumption queues polled by multiple consumption threads, the batch number identifier is used for distinguishing each piece of data change information, rollback operation of single piece of data is facilitated, and then the data change information after the batch number identifier is bound is stored in a memory.
In the technical scheme disclosed in this embodiment, the data change information is sent to a consumption queue as a consumption event, the data change information in the consumption queue is polled based on a plurality of consumption threads, a lot number identifier is bound to the data change information, and the data change information after the lot number identifier is bound is stored in a memory. In this way, the binary log is rapidly analyzed in a production-consumer mode, and the data change information in the consumption queue is polled based on a plurality of consumption threads, so that the throughput performance of log consumption is improved, and the data rollback efficiency is improved.
Further, before the step of polling the data change information in the consumption queue based on the plurality of consumption threads and binding the lot number identification for the data change information, the method further includes:
acquiring a transaction identifier corresponding to a current transaction of the database;
and carrying out association binding on the batch number identification corresponding to the change data in the current transaction and the transaction identification.
In this embodiment, the transaction identifier may be set to distinguish the data change information in the execution process of each transaction, so as to facilitate rollback operation in units of transactions. Before the step of binding the batch number identifiers for the data change information, the transaction identifier corresponding to the current transaction of the database is obtained and is also a global transaction identifier, and the transaction identifier is used for binding the batch number identifier corresponding to the change data in the current transaction with the transaction identifier based on the transaction rollback operation, so that when a rollback instruction corresponding to the transaction identifier is received, the corresponding batch number identifier can be indexed through the transaction identifier, and rollback of the change data under one transaction is completed.
Therefore, the transaction identifier is bound with the batch number identifier corresponding to the change data in the transaction, so that the data rollback taking the transaction as a unit can be realized, and the data rollback efficiency is improved.
A specific application scenario is provided below, with reference to fig. 6:
the production end is binlog client and is used for collecting logs synchronized by a master of a database and obtaining global transaction ID, wherein the ID is used for rolling back based on the transaction. After the production end analyzes the data, data change information is transmitted to a local consumption queue Disrupter, and the throughput performance of log consumption is improved. The local consumption queue trains the data change information of the consumption production end, marks batch numbers for the data change information of each line of change data of each database table, and the batch ids are used for rolling back operation based on single data.
Referring to FIG. 7, mysql generates Gtid for a transaction that manipulates the data changes of multiple tables, the data change information for each row of change data for each database table is bound to the batch Id, and the fields of the record table change data back and forth, thus creating the following requirements: binding a plurality of batch Ids under Gtid, which are used for realizing data rollback of a transaction; one patchId implements rollback of single form single line change data.
Further, the step of determining the target change data corresponding to the rollback instruction includes:
determining a target transaction identifier according to the rollback instruction, determining a batch number identifier which is associated and bound with the target transaction identifier, and taking change data corresponding to the batch number identifier as the target change data; or alternatively
And determining a batch number identification according to the rollback instruction, and taking change data corresponding to the batch number identification as the target change data.
In this embodiment, since the transaction identifiers are all bound with a plurality of lot number identifiers, each lot number identifier is bound with data change information corresponding to a plurality of single change data, the rollback instruction can specify a target transaction identifier or a lot number identifier that needs to be rolled back, so as to implement data rollback on the target transaction or the target change data, specifically, determine the target transaction identifier according to the rollback instruction, determine the lot number identifier associated with the target transaction identifier, and use the data change information corresponding to the lot number identifier as the target data change information, or determine the lot number identifier according to the rollback instruction, and use the data change information corresponding to the lot number identifier as the target data change information.
For example, when the user triggers the rollback instruction, a transaction rollback, namely, gtid rollback, may be selected, or the rollback may be performed according to data change information of a certain line of a certain table, namely, the batch id rollback, and the parsing process is as follows: selecting a rollback mode, if the rollback mode is based on transaction rollback, the Gtid is associated with a plurality of batch Ids, and inquiring data change information of a certain line of a certain table according to the batch Ids; if the batch rollback is based, the data change information of a certain line of a certain table is directly inquired according to the batch Id.
Wherein for delete operation: acquiring data before data change of each field according to the data change information, and further reversely generating the sql of insert, namely: insert into xx tables (field 1, field 2) values (values before field 1 change, values before field 2 change);
for insert (add) operations: acquiring data after each field change according to the data change information, and binding and associating the fields, the primary keys and the field contents, wherein the delete sql is deleted reversely based on the primary keys, such as delete from xx table where primary key= 'xxx';
for update operation: and acquiring the data before the change of each field according to the data change information, and binding the fields, the primary keys and the field contents, and reversely generating an update sql based on the primary key update, for example, the update xx table set 1 = the value before the change of the field 1, and the field 2 = the value before the change of the field 2, wherein the primary key= 'xxx' is the value before the change of the field 2.
All sql is generated reversely, a database execution transaction is started, the sql is executed, and the rollback of the data is completed. In the rollback operation process, the database is not required to execute binlog rollback, the business stop risk is avoided, and the data change history can be formed for auditing.
Thus, the rollback of the transaction or the rollback of single changed data is realized through the rollback instruction, and the data rollback efficiency is improved.
Further, after the step of monitoring the binary log of the database and determining the data change information corresponding to the change data in the database according to the binary log, the method further includes:
determining information to be displayed corresponding to each change data according to the data change information;
generating a graphical interface according to the information to be displayed;
and receiving a rollback instruction of a user based on the graphical interface.
In this embodiment, in order to facilitate the user to select the target data change information that needs to be rolled back, a graphical operation interface of a user view is required to be provided, firstly, the data change information that can be rolled back or the data change information that can be rolled back corresponding to the current login user is determined, then, according to the data change information, the information to be displayed corresponding to each piece of the data change that can be rolled back is determined, including a database of the changed data, a log recording time, a changed target field, a field value, a comparison before and after the data change, a DML type, and the like, and a graphical interface that can be rolled back for a certain piece of the changed data or for a certain transaction is generated according to the information to be displayed, and referring to fig. 8, the graphical interface can also provide a data change history for analyzing, identifying and auditing all the operation conditions from newly added, updated to deleted.
Therefore, the user can conveniently screen out the data which is wanted to be operated to execute the rollback operation, the user can conveniently give the rollback instruction, and the data rollback efficiency can be improved.
In addition, the embodiment of the invention also provides a data rollback device, which comprises a memory, a processor and a data rollback program stored on the memory and capable of running on the processor, wherein the data rollback program realizes the steps of the data rollback method in each embodiment when being executed by the processor.
In addition, the embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a data rollback program, and the data rollback program realizes the steps of the data rollback method in the above embodiments when being executed by a processor.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising several instructions for causing a data rollback apparatus to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. A data rollback method, the data rollback method comprising:
monitoring a binary log of a database, and determining data change information corresponding to change data in the database according to the binary log;
when a rollback instruction is received, determining target data change information corresponding to the rollback instruction;
and reversely generating a rollback code according to the target data change information, and executing the rollback code.
2. The data rollback method of claim 1 wherein the step of monitoring a binary log of a database and determining data change information corresponding to change data in the database based on the binary log comprises:
deploying log monitoring application based on the application instance of the container management platform;
triggering a ready probe to fight against the distributed lock when the application instance is ready;
when the race is successful, initializing a configuration item of the log monitoring application based on the application instance, and monitoring the binary log based on the configuration item;
and triggering a survival probe in the running process of the application instance, and renewing the distributed lock so that the log monitoring application continuously monitors the binary log.
3. The method of rolling back data according to claim 2, wherein after the step of triggering a survival probe during the running of the application instance to renew the distributed lock so that the log listening application continues to listen to the binary log, further comprising:
triggering a stop probe when the stop condition of monitoring the binary log is met, determining the current monitored log position of the log monitoring application based on the stop probe, and stopping the current application instance;
when the next application instance is ready, taking the log position as a monitoring starting position corresponding to a log monitoring application in the next application instance;
wherein the stop condition of the listening archive log comprises at least one of:
the application instance fails, restarts, redeploys;
and receiving a stop instruction for monitoring the archive log.
4. The data rollback method of claim 2 wherein the step of initializing a configuration item of the log listening application based on the application instance comprises:
acquiring a database object set, fields, field types and positions of fields in a table of each data table in the database;
and taking the database object set, the fields, the field types and the positions of the fields in a data table as the configuration items.
5. The method for rolling back data according to claim 1, wherein the step of monitoring a binary log of a database and determining data change information corresponding to change data in the database according to the binary log further comprises:
sending the data change information to a consumption queue as a consumption event;
polling data change information in the consumption queue based on a plurality of consumption threads, and binding a batch number identifier for the data change information;
and storing the data change information after the batch number identification is bound in a memory.
6. The method of claim 5, wherein prior to the step of polling the consuming queues for data change information based on a plurality of consuming threads and binding a lot number identification for the data change information, further comprising:
acquiring a transaction identifier corresponding to a current transaction of the database;
and carrying out association binding on the batch number identification corresponding to the change data in the current transaction and the transaction identification.
7. The method of data rollback as recited in claim 6 wherein said step of determining target change data corresponding to said rollback instruction comprises:
determining a target transaction identifier according to the rollback instruction, determining a batch number identifier which is associated and bound with the target transaction identifier, and taking data change information corresponding to the batch number identifier as the target data change information; or alternatively
And determining a batch number identification according to the rollback instruction, and taking data change information corresponding to the batch number identification as the target data change information.
8. The method for rolling back data according to claim 7, wherein the step of monitoring a binary log of a database and determining data change information corresponding to change data in the database according to the binary log further comprises:
determining information to be displayed corresponding to each change data according to the data change information;
generating a graphical interface according to the information to be displayed;
and receiving a rollback instruction of a user based on the graphical interface.
9. A data rollback apparatus, the data rollback apparatus comprising: memory, a processor and a data rollback program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the data rollback method according to any of claims 1 to 8.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a data rollback program, which when executed by a processor, implements the steps of the data rollback method according to any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310783735.5A CN116841983A (en) | 2023-06-28 | 2023-06-28 | Data rollback method, device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310783735.5A CN116841983A (en) | 2023-06-28 | 2023-06-28 | Data rollback method, device and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116841983A true CN116841983A (en) | 2023-10-03 |
Family
ID=88162829
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310783735.5A Pending CN116841983A (en) | 2023-06-28 | 2023-06-28 | Data rollback method, device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116841983A (en) |
-
2023
- 2023-06-28 CN CN202310783735.5A patent/CN116841983A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10769001B2 (en) | System and method for process state processing | |
| CN107220142B (en) | Method and device for executing data recovery operation | |
| CN108563502B (en) | Task scheduling method and device | |
| CN110895484A (en) | Task scheduling method and device | |
| CN110895487B (en) | Distributed task scheduling system | |
| CN110895488B (en) | Task scheduling method and device | |
| US10204012B2 (en) | Impact analysis-based task redoing method, impact analysis calculation apparatus, and one-click resetting apparatus | |
| CN111324423A (en) | Method and device for monitoring processes in container, storage medium and computer equipment | |
| CN110895483A (en) | Task recovery method and device | |
| CN110196749B (en) | Virtual machine recovery method and device, storage medium and electronic device | |
| CN106874343B (en) | Data deletion method and system for time sequence database | |
| CN110895486A (en) | Distributed task scheduling system | |
| CN113900842A (en) | Message consumption method and device, electronic equipment and computer storage medium | |
| CN113342839A (en) | Data processing method and device, terminal equipment and storage medium | |
| CN109815233A (en) | A kind of processing method and system of tables of data | |
| EP3377970B1 (en) | Multi-version removal manager | |
| CN116841983A (en) | Data rollback method, device and computer readable storage medium | |
| JP5943753B2 (en) | Virtual machine management system, virtual machine management method and program | |
| CN114675950A (en) | Task scheduling method and device | |
| JP2001356946A (en) | Method, device for executing work flow and recording medium in which work flow execution program is recorded | |
| CN116340051A (en) | Data processing method, related device and equipment | |
| CN110011832B (en) | Configuration issuing method and device for planned tasks | |
| CN108958827B (en) | Method and system for processing program upgrading fault | |
| CN108206933B (en) | Video data acquisition method and device based on video cloud storage system | |
| KR970007625A (en) | How to Manage Multiple Application Processes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |