CN115145831A

CN115145831A - Non-invasive test data recovery method and system

Info

Publication number: CN115145831A
Application number: CN202211076781.3A
Authority: CN
Inventors: 鲁健翔; 徐轲; 薛海伟; 戴小村
Original assignee: Hunan Changyin May 8th Consumer Finance Co ltd
Current assignee: Hunan Changyin May 8th Consumer Finance Co ltd
Priority date: 2022-09-05
Filing date: 2022-09-05
Publication date: 2022-10-04
Anticipated expiration: 2042-09-05
Also published as: CN115145831B

Abstract

The disclosure relates to a non-intrusive test data recovery method and system. The method comprises the following steps: intercepting a data modification statement in a test request sent to a database by an application system when a plurality of test cases are executed in sequence, and generating a data rollback operation statement corresponding to the data modification statement; establishing a mapping relation, wherein the mapping relation comprises a case number of each test case, a data modification statement corresponding to each test case, and a mapping relation between data rollback operation statements corresponding to the data modification statement corresponding to each test case; determining a data modification statement corresponding to a corresponding target test case or a data rollback operation statement corresponding to a data modification statement corresponding to a target test case based on the case number and the mapping relation of the specified test case; and executing the data modification statement corresponding to the target test case, or executing the data rollback operation statement corresponding to the data modification statement of the target test case to realize data recovery.

Description

Non-invasive test data recovery method and system

Technical Field

The embodiment of the disclosure relates to the technical field of computers, in particular to a non-intrusive test data recovery method and system.

Background

At present, in a test stage of an application system, a tester often needs to construct test data according to a scene in a test case, and needs to recover to an initial state or recover to an intermediate state in which part of the test cases are completely executed to retest after the test case is executed, which relates to data recovery operation on a database used by the application system.

In the related art, when an application system is tested, an SQL script is prepared in advance, all data required by a test case is generally required to be prepared into the SQL script before the test is executed, and when different test cases are executed and data needs to be recovered, the SQL script corresponding to the test case is executed. However, the cost of manually maintaining the SQL script is too high, a great amount of time is consumed for preparing script data under a complex scene, and testers need to know the database table structure of the application system in advance, so that a certain implementation threshold is achieved, and the testing efficiency is low.

In other related technologies, data recovery may be achieved by backing up and restoring a database, but the entire database needs to be backed up in time after a test case is executed each time, and a corresponding database backup file is restored when data recovery is required. The technical scheme has the defects that the physical storage space is occupied, in addition, the backup and the recovery operation need to be executed manually, the backup operation is easy to miss or the recovery operation is easy to miss, and the overall testing efficiency is low.

Disclosure of Invention

To solve the above technical problem or at least partially solve the above technical problem, embodiments of the present disclosure provide a method and system for non-intrusive test data recovery.

In a first aspect, an embodiment of the present disclosure provides a method for recovering non-intrusive test data, including:

intercepting one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent to a database by an application system when a plurality of test cases are executed in sequence, and correspondingly generating data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data updating statement; each test request at least carries a case number of a corresponding test case and one or more of a data insertion statement, a data deletion statement and a data updating statement, and the case number of each test case is used for uniquely identifying the test case and indicating the execution sequence of the test case;

establishing a mapping relation, wherein the mapping relation comprises a case number of each test case, one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to each test case, and a mapping relation between data rollback operation statements corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to each test case;

based on the case number of the specified test case and the mapping relation, determining one or more of a data insertion statement, a data deletion statement and a data update statement corresponding to the corresponding target test case, or one or more data rollback operation statements respectively corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to the target test case; wherein the specified test case is any one of the plurality of test cases;

and executing one or more of the data insertion statement, the data deletion statement and the data update statement corresponding to the target test case, or executing one or more of the data insertion statement, the data deletion statement and the data update statement corresponding to the target test case to realize data recovery.

In an embodiment, the generating the data rollback operation statement corresponding to each of the data insertion statement, the data deletion statement, and the data update statement includes:

generating a first data rollback operation statement corresponding to the data insertion statement, wherein the first data rollback operation statement is a target data deletion statement, and data to be deleted by the target data deletion statement is the same as data to be inserted by the data insertion statement;

generating a second data rollback operation statement corresponding to the data deletion statement, wherein the second data rollback operation statement is a target data insertion statement, and data to be inserted by the target data insertion statement is the same as data to be deleted by the data deletion statement;

and generating a third data rollback operation statement corresponding to the data update statement, wherein the third data rollback operation statement is a target data update statement, and the data to be updated by the target data update statement is the data before the data update statement performs the data update operation.

In one embodiment, the intercepting application system sends one or more of a data insertion statement, a data deletion statement and a data update statement in a test request to a database when a plurality of test cases are executed in sequence, and the intercepting application system includes:

intercepting one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent by each test case to the database based on an agent program; wherein the agent is implemented based on Java language and introduced into the application system in the form of JAR package.

In one embodiment, before one or more of a data insertion statement, a data deletion statement and a data update statement in a test request sent by the intercepting application system to the database when a plurality of test cases are executed in sequence, the method includes:

modifying a first configuration file of the database to start a binary log function; wherein the first configuration file is used for opening or closing a binary log function of the database;

establishing a first data table in the database;

after intercepting one or more of a data insertion statement, a data deletion statement and a data update statement in each test request, before executing one or more of the data insertion statement, the data deletion statement and the data update statement in each test request, inserting a case number of a test case corresponding to each corresponding test request into the first data table, so as to automatically record the insertion operation of the first data table in a binary log of the database.

In one embodiment, before one or more of a data insertion statement, a data deletion statement, and a data update statement in a test request sent by the intercepting application system to the database when the plurality of test cases are executed in sequence, the method includes:

modifying a second configuration file of the database to open a database global transaction ID; wherein the second configuration file is used for opening or closing a global transaction ID function of the database;

establishing a second data table in the database;

the generating of the data rollback operation statement corresponding to each of the data insertion statement, the data deletion statement, and the data update statement includes:

analyzing the binary log to obtain at least one executable SQL statement and generating a data rollback SQL statement corresponding to each SQL statement; the at least one SQL statement is one or more related SQL statements in a data insertion statement, a data deletion statement and a data updating statement;

the establishing of the mapping relation comprises the following steps:

analyzing the binary log to obtain a global transaction ID of at least one recorded transaction, and determining one or more SQL statements submitted by the transaction indicated by each global transaction ID based on the global transaction ID of the at least one transaction;

judging whether one or more SQL statements submitted by the affair indicated by each global affair ID have the insertion operation in the first data table, if so, determining the corresponding global affair ID as a target global affair ID, and acquiring the case number of the test case corresponding to the affair indicated by the target global affair ID;

establishing a first mapping relation between the use case number of the test case corresponding to the transaction indicated by the target global transaction ID and the target global transaction ID, simultaneously establishing a second mapping relation between the target global transaction ID and one or more SQL statements submitted by the transaction indicated by the target global transaction ID and data rollback SQL statements corresponding to the one or more SQL statements, establishing the mapping relation based on the first mapping relation and the second mapping relation, and writing the mapping relation into the second data table.

In one embodiment, before parsing the binary log, the method further comprises:

acquiring a binary log of the database from a main node of the database based on a slave node simulation device in a preset listener;

analyzing the binary log to obtain at least one executable SQL statement, and generating a data rollback SQL statement corresponding to each SQL statement, wherein the data rollback SQL statement comprises the following steps:

and the slave node simulation device analyzes the binary log to obtain at least one executable SQL statement and generates a data rollback SQL statement corresponding to each SQL statement.

In one embodiment, the establishing a mapping relationship includes: and establishing the mapping relation based on a mapping device in the listener.

In a second aspect, an embodiment of the present disclosure provides a system for recovering non-invasive test data, including:

the system comprises an interception construction module, a data retrieval module and a data retrieval module, wherein the interception construction module is used for intercepting one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent to a database by an application system when a plurality of test cases are executed in sequence, and correspondingly generating data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data updating statement; each test request at least carries a case number of a corresponding test case and one or more of a data insertion statement, a data deletion statement and a data updating statement, and the case number of each test case is used for uniquely identifying the test case and indicating the execution sequence of the test case;

the mapping relation establishing module is used for establishing a mapping relation, wherein the mapping relation comprises a case number of each test case, one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to each test case, and the mapping relation among one or more data rollback operation statements respectively corresponding to the data insertion statement, the data deletion statement and the data updating statement corresponding to each test case;

the data recovery preparation module is used for determining one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to a corresponding target test case based on the case number of the specified test case and the mapping relation, or determining one or more data rollback operation statements corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to the target test case; wherein the specified test case is any one of the plurality of test cases;

and the data recovery execution module is used for executing one or more of the data insertion statement, the data deletion statement and the data update statement corresponding to the target test case, or executing one or more data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data update statement of the target test case to realize data recovery.

In a third aspect, embodiments of the present disclosure provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the non-intrusive test data recovery method according to any of the above embodiments.

In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including:

a processor; and

a memory for storing a computer program;

wherein the processor is configured to perform the non-intrusive test data recovery method of any of the above embodiments via execution of the computer program.

Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:

the non-intrusive test data recovery method and system provided by the embodiment of the disclosure intercept one or more of a data insertion statement, a data deletion statement and a data update statement in a test request sent by an application system to a database when a plurality of test cases are executed in sequence, and correspondingly generate data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data update statement; each test request at least carries a case number of a corresponding test case and one or more of a data insertion statement, a data deletion statement and a data updating statement; establishing a mapping relation, wherein the mapping relation comprises a case number of each test case, one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to each test case, and a mapping relation between data rollback operation statements corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to each test case; determining one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to a corresponding target test case based on a case number of a specified test case and the mapping relation, or determining a data rollback operation statement corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to the target test case; and executing one or more of the data insertion statement, the data deletion statement and the data update statement corresponding to the target test case, or executing one or more of the data insertion statement, the data deletion statement and the data update statement corresponding to the target test case to realize data recovery. Thus, based on the established mapping relationship, when it is necessary to restore the data state corresponding to any test case, that is, the specified test case, it is only necessary to execute one or more of the forward or reverse SQL statement, that is, the data insertion statement, the data deletion statement, and the data update statement associated with the test case, or the corresponding data rollback operation statement, and a tester does not need to know the database table structure of the application system in advance, and does not need to maintain a large amount of SQL scripts, and can operate only by remembering the case number of the test case. According to the case numbers of the test cases, the data state after any test case is executed can be quickly and flexibly restored and restored, meanwhile, the storage space wasted by backing up the whole database is avoided, misoperation possibly caused by restoring the database is avoided, the data restoration can be integrally and quickly and flexibly carried out, the defects that the whole database is backed up and restored and an SQL script is prepared and maintained in advance in the existing scheme are avoided, and the overall test efficiency is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the embodiments or technical solutions in the prior art description will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.

FIG. 1 is a flow chart of a method for non-intrusive test data recovery in accordance with an embodiment of the present disclosure;

FIG. 2 is a flow chart of a non-intrusive test data recovery method according to another embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a non-intrusive test data recovery system according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a non-intrusive test data recovery system according to an embodiment of the present disclosure;

fig. 5 is a schematic diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.

It is to be understood that, hereinafter, "at least one" means one or more, "a plurality" means two or more. "and/or" is used to describe the association relationship of the associated objects, meaning that there may be three relationships, for example, "a and/or B" may mean: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single item(s) or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.

Fig. 1 is a flowchart of a method for non-intrusive test data recovery according to an embodiment of the present disclosure, where the method may be executed on a computer or a server, and specifically includes the following steps:

step S101: intercepting one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent to a database by an application system when a plurality of test cases are executed in sequence, and correspondingly generating data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data updating statement; each test request at least carries a case number of a corresponding test case and one or more of a data insertion statement, a data deletion statement and a data updating statement, and the case number of each test case is used for uniquely identifying the test case and indicating the execution sequence of the test case.

The application system may be, for example, a mail system, an online editing system, an application program, and the like, which is not limited in this embodiment. In the development and test stage, the application system executes the test case to perform a test, such as a function, which may involve an operation on the database, such as modifying some data in the database, but is not limited thereto. The database may be a MySQL database, but is not so limited. The application system may send a test request to the database when executing a certain test case, where the test request may carry a case number of the certain test case and any one or more of a data Insert (Insert) statement, a data Delete (Delete) statement, or a data Update (Update) statement for modifying certain data in the database, and the statements may specifically be SQL statements.

In this embodiment, any one or more of a data insertion (Insert) statement, a data deletion (Delete) statement, or an data Update (Update) statement in a test request sent by each test case is intercepted, and then a data rollback operation statement, that is, a reverse SQL statement (also called a rollback SQL statement), corresponding to each of the intercepted data insertion statement, data deletion statement, and data Update statement, that is, a forward SQL statement, is generated.

Step S102: and establishing a mapping relation, wherein the mapping relation comprises a case number of each test case, one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to each test case, and a mapping relation among data rollback operation statements corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to each test case.

Illustratively, after a plurality of test cases are executed, the final mapping relationship is established. Specifically, for each test case, a mapping relationship is established between a case number corresponding to the test case, one or more of the intercepted data insertion statement, the intercepted data deletion statement and the data update statement, and one or more data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data update statement. That is, the case number of each test case is associated with a plurality of SQL statements submitted by the test case and a corresponding plurality of rollback SQL statements.

Step S103: and determining one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to the corresponding target test case based on the case number of the specified test case and the mapping relation, or determining a data rollback operation statement corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to the target test case. Wherein the specified test case is any one of the plurality of test cases.

For example, the specified test case is any one of the multiple test cases, so the solution of this embodiment may recover to the data state after the execution of any one test case is completed. For example, 10 test cases (with case numbers of 01-10 in sequence, that is, the execution sequence is 01-10) are sequentially executed, when the data state before the 6 th test case is executed, that is, after the 5 th test case is executed needs to be restored, it is determined that the corresponding target test case is the test case with the case number of 10-06, and then all the reverse SQL statements corresponding to the case numbers of 10-06, that is, the reverse SQL statements, which are data rollback operation statements corresponding to one or more of the data insertion statements, the data deletion statements, and the data update statements submitted by the 10 th-06 test cases, can be determined based on the mapping relationship.

Step S104: and executing one or more of the data insertion statement, the data deletion statement and the data update statement corresponding to the target test case, or executing one or more of the data insertion statement, the data deletion statement and the data update statement corresponding to the target test case to realize data recovery.

Illustratively, the data can be recovered by executing all the reverse SQL statements related to the case numbers of 10-06 in reverse order. On the basis of the scheme, for example, when the data state after the 8 th test case is executed needs to be restored, the corresponding target test case is determined to be the test case with the case number of 07-08, all forward SQL statements corresponding to the case numbers of 07-08 can be determined based on the mapping relation, and then all the forward SQL statements related to the case numbers of 07-08 are sequentially executed, so that the data can be restored.

Based on the established mapping relationship, when the data state corresponding to any test case, that is, the specified test case, needs to be restored, only one or more of the forward or reverse SQL statement, that is, the data insertion statement, the data deletion statement, and the data update statement associated with the test case, or the corresponding data rollback operation statement is executed, and a tester does not need to know the database table structure of the application system in advance, does not need to maintain a large amount of SQL scripts, and can operate only by remembering the case number of the test case. According to the case numbers of the test cases, the data state after any test case is executed can be quickly and flexibly restored and restored, meanwhile, the storage space wasted by backing up the whole database is avoided, misoperation possibly caused by restoring the database is avoided, the data restoration can be integrally and quickly and flexibly carried out, the defects that the whole database is backed up and restored and an SQL script is prepared and maintained in advance in the existing scheme are avoided, and the overall test efficiency is improved.

In an embodiment, the generating of the data rollback operation statement corresponding to each of the data insertion statement, the data deletion statement, and the data update statement in step S101 may specifically include: generating a first data rollback operation statement corresponding to the data insertion statement, wherein the first data rollback operation statement is a target data deletion statement, and data to be deleted by the target data deletion statement is the same as data to be inserted by the data insertion statement; generating a second data rollback operation statement corresponding to the data deletion statement, wherein the second data rollback operation statement is a target data insertion statement, and data to be inserted by the target data insertion statement is the same as data to be deleted by the data deletion statement; and generating a third data rollback operation statement corresponding to the data update statement, wherein the third data rollback operation statement is a target data update statement, and the data to be updated by the target data update statement is the data before the data update statement performs the data update operation.

Illustratively, the above scheme is that an insert statement generates a delete statement, a delete statement generates an insert statement, and an update statement generates a reverse update statement. Here again, for example: assuming that a record exists in a data table user of the database, and id =1 name = zhang, the specific generation method is as follows:

1) Insert operation

Forward SQL statement insert inter user values (1, 'Zhang three');

the reverse SQL statement delete from user where id =1;

2) Delete operation

Forward SQL statement delete from user where id =1;

inverse SQL statement insert inter user values (1, 'Zhang three');

3) Update operation

Forward SQL statement update user set name = 'lie four' where id =1;

the inverse SQL statement update user set name = 'three' where id =1.

In an embodiment, the intercepting, in step S101, one or more of a data insertion statement, a data deletion statement, and a data update statement in a test request sent by an application system to a database when a plurality of test cases are executed in sequence may specifically include: intercepting one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent by each test case to the database based on an agent program; wherein the agent is implemented based on Java language and introduced into the application system in the form of JAR package.

Specifically, as shown in fig. 3, in this embodiment, a proxy program mysqlisnpectagent is implemented in Java language, and the proxy program mysqlisnpectagent is introduced into the application system in the form of JAR package, and it is not necessary to modify any code of the application system itself, that is, a non-intrusive test data recovery scheme is implemented, so that potential influence on test data recovery caused by modifying the code of the application system itself is avoided, and reliability of test data recovery is improved.

In one embodiment, before intercepting one or more of a data insertion statement, a data deletion statement and a data update statement in a test request sent by an application system to a database when a plurality of test cases are executed in sequence in step S101, the method includes:

step a): modifying a first configuration file of the database to start a binary log function; wherein the first configuration file is used for opening or closing a binary log function of the database.

For example, in this embodiment, a function related to a Binary Log (Binary Log) of the MySQL database needs to be started, and this function is mainly used to record all data operation statements, such as a data insertion statement, a data deletion statement, and a data update statement, that actually perform data modification on the MySQL database. The operation of turning on the binary journal function is as follows: and modifying the configuration file my.cnf of the MySQL database of the test environment, which comprises the following specific steps:

log-bin=mysql-bin

binlog_format=row

server-id=1。

step b): a first data table is established in the database.

For example, a data table with a table name binlog _ rela is built in the MySQL database.

Step c): after intercepting one or more of a data insertion statement, a data deletion statement and a data update statement in each test request, before executing one or more of the data insertion statement, the data deletion statement and the data update statement in each test request, inserting a case number of a test case corresponding to each corresponding test request into the first data table, so as to automatically record the insertion operation of the first data table in a binary log of the database.

That is, before one or more of a data insert statement, a data delete statement, and a data update statement in a test request sent by each test case reaches the database and is executed, the insert, update, and delete operations are intercepted, and the use case numbers of the test cases initiating the operations are inserted into the first data table binlog _ rela, so that all insert, update, and delete operations automatically record the insertion operation of a binlog _ rela table in the Binary Log of the MySQL database.

On the basis of the foregoing embodiment, in an embodiment, before intercepting one or more of a data insertion statement, a data deletion statement, and a data update statement in a test request sent by an application system to a database when a plurality of test cases are executed in sequence in step S101, the method includes:

step d): modifying a second configuration file of the database to open a database Global Transaction ID (GTID); wherein the second configuration file is used for opening or closing a global transaction ID function of the database.

Illustratively, the GTID correlation function of the MySQL database is started, and the GTID is a global transaction ID, which can ensure that each transaction submitted on the master library has a unique ID in the cluster. The specific operation of opening is as follows: and modifying a configuration file my.cnf of the MySQL database of the test environment:

gtid_mode=ON

enforce-gtid-consistency=ON。

it will be appreciated that step d) may be performed simultaneously with step a) described above.

Step e): and establishing a second data table in the database.

For example, a data table with a table name binlog _ sql _ record is established in the MySQL database as the second data table. It will be appreciated that step e) may also be performed simultaneously with step b) described above.

Illustratively, the binlog _ sql _ record table contains field names as follows:

id-Primary Key

Biz _ seq-case number of test case

Forward _ SQL forward SQL statements

rollback _ SQL reverse SQL statement

GTID-GTID code

create _ time-creation time.

Accordingly, in step S101, data rollback operation statements corresponding to the data insertion statement, the data deletion statement, and the data update statement are generated, the method specifically comprises the following steps: analyzing the binary log to obtain at least one executable SQL statement and generating a data rollback SQL statement corresponding to each SQL statement; the at least one SQL statement is one or more related SQL statements of a data insertion statement, a data deletion statement, and a data update statement. For this part of the content, reference may be made to the related description in the foregoing embodiment, and details are not repeated here, except that all executable SQL statements, i.e., SQL statements corresponding to insert, update, and delete operations, of the log record are obtained by parsing from the binary log.

With reference to fig. 2, the establishing of the mapping relationship in step S102 may specifically include the following steps:

step S201: and analyzing the binary log to obtain a global transaction ID of at least one recorded transaction, and determining one or more SQL statements submitted by the transaction indicated by each global transaction ID based on the global transaction ID of the at least one transaction.

Illustratively, the Binary Log is parsed line by line to find statements beginning with SET @ session GTID _ NEXT to obtain GTID of one or more transactions recorded by the Log, and then to determine one or more SQL statements submitted by the one or more transactions recorded by the Log. Specifically, a mapping between different GTIDs and corresponding SQL statement sets may be established, where an SQL statement set includes one or more SQL statements submitted by the transaction indicated by the corresponding GTID.

Step S202: and judging whether one or more SQL statements submitted by the transaction indicated by each global transaction ID have the insertion operation in the first data table, if so, determining the corresponding global transaction ID as a target global transaction ID, and acquiring the case number of the test case corresponding to the transaction indicated by the target global transaction ID.

Illustratively, for a transaction indicated by each GTID of a log record, it is determined whether one or more SQL statements in the SQL statement set corresponding to the transaction indicated by each GTID have the insertion operation in the first data table, that is, it is determined whether a record of the insertion operation of one or more SQL statements in the SQL statement set in the first data table is recorded in the log, if yes, it is determined that the transaction indicated by the GTID corresponding to the SQL statement set is a transaction submitted and executed by a test case rather than a transaction submitted and executed by a non-test case, the GTID corresponding to the SQL statement set is used as a target GTID, and a case number of the test case corresponding to the transaction indicated by the target GTID, that is, a case number of the test case submitting the transaction executed by the target GTID, is obtained. It will be appreciated that the transactions recorded in the binary log are not all transactions for which test cases committed execution.

Step S203: establishing a first mapping relation between the use case number of the test case corresponding to the transaction indicated by the target global transaction ID and the target global transaction ID, simultaneously establishing a second mapping relation between the target global transaction ID and one or more SQL statements submitted by the transaction indicated by the target global transaction ID and data rollback SQL statements corresponding to the one or more SQL statements, establishing the mapping relation based on the first mapping relation and the second mapping relation, and writing the mapping relation into the second data table.

Illustratively, after obtaining a case number of a test case that has submitted a transaction indicated by a target GTID, a first mapping relationship between the case number of the test case corresponding to the transaction indicated by the target GTID and the target GTID is established, and at the same time, a second mapping relationship between one or more SQL statements (i.e., forward SQL statements) submitted by the transaction indicated by the target GTID and data rollback SQL statements (i.e., reverse SQL statements) corresponding to the one or more SQL statements is established, and the mapping relationship is established based on the first mapping relationship and the second mapping relationship and written into a second data table, such as a binlog _ SQL _ record table, for storage. Specifically, a final mapping relation is established by taking the target GTID in the first mapping relation and the second mapping relation as a coupling condition, and the final mapping relation includes a mapping relation between a case number of a test case corresponding to the transaction indicated by the target GTID and a corresponding forward SQL statement and a reverse SQL statement.

The data rollback SQL statement (i.e., the reverse SQL statement) corresponding to one or more SQL statements (i.e., the forward SQL statement) submitted by the transaction indicated by the target GTID is determined in the step S101 of analyzing the binary log to obtain at least one executable SQL statement and generating the data rollback SQL statement corresponding to each SQL statement. That is, the forward SQL statement and the reverse SQL statement that parse the binary log construct can be regarded as SQL statement library, and the forward SQL statement and the corresponding data rollback SQL statement (i.e., reverse SQL statement) submitted by the transaction indicated by the target GTID are partial SQL statements determined from the SQL statement library.

In the embodiment, the global transaction ID obtained by analyzing the binary log and the one or more SQL statements submitted by the transaction indicated by the global transaction ID are analyzed, whether the one or more SQL statements submitted by the transaction indicated by the global transaction ID in the log have an insert operation in the first data table is judged to accurately determine the target global transaction ID, and the target global transaction ID is used as an intermediate bridge to establish the mapping relationship.

In an embodiment, before parsing the binary log in the foregoing embodiments, the method may further include: and acquiring the binary log of the database from the main node of the database based on a slave node simulation device in a preset listener.

For example, as shown in fig. 3, in this embodiment, a listener MySQLBinLogListener may be implemented in Java language, and the listener includes a slave node simulation device, and the slave node simulation device is responsible for sending a dump protocol request to a master node of a database. After receiving the dump protocol request, the master node starts to push the Binary Log to the slave node simulation device.

Correspondingly, in the step S101, the analyzing the binary log to obtain at least one executable SQL statement, and generating a data rollback SQL statement corresponding to each SQL statement includes: and the slave node simulation device analyzes the binary log to obtain at least one executable SQL statement and generates a data rollback SQL statement corresponding to each SQL statement.

That is, the slave node simulation apparatus parses the Binary Log into executable SQL statements (forward SQL statements) and generates corresponding reverse SQL statements (rollback SQL statements), that is, an insert statement generates a delete statement, a delete statement generates an insert statement, and an update statement generates a reverse update statement, which may refer to the detailed description in the foregoing embodiments.

Therefore, in this embodiment, the slave node simulation apparatus based on the listener can quickly and conveniently obtain the binary log and analyze the binary log to construct the forward SQL statement and the reverse SQL statement, so as to indirectly improve the efficiency of establishing the subsequent mapping relationship, so that the establishment of the mapping relationship can be quickly completed, and the overall testing efficiency is indirectly further improved.

Based on the foregoing embodiment, in an embodiment, the establishing of the mapping relationship in the foregoing embodiment may be specifically established based on a mapping apparatus in the listener.

That is to say, as shown in fig. 3, the listener MySQLBinLogListener may include two parts, namely, a slave node simulator and a mapping device, where the mapping device is responsible for establishing a mapping relationship between a forward SQL statement and a reverse SQL statement, which are obtained by parsing in the slave node simulator, and a GTID and a use case number of a test case.

In this embodiment, the listener is used to establish the mapping relationship, and the listener includes the above two parts, so that the mapping relationship can be efficiently and quickly established in a cooperative manner, thereby indirectly further improving the overall testing efficiency.

It should be noted that although the various steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that these steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc. Additionally, it will also be readily appreciated that the steps may be performed synchronously or asynchronously, e.g., among multiple modules/processes/threads.

As shown in fig. 4, an embodiment of the present disclosure provides a system for recovering non-invasive test data, including:

the interception construction module 401 is configured to intercept one or more of a data insertion statement, a data deletion statement, and a data update statement in a test request sent by an application system to a database when a plurality of test cases are executed in sequence, and generate corresponding data rollback operation statements of the data insertion statement, the data deletion statement, and the data update statement; each test request at least carries a case number of a corresponding test case and one or more of a data insertion statement, a data deletion statement and a data updating statement, and the case number of each test case is used for uniquely identifying the test case and indicating the execution sequence of the test case;

a mapping relationship establishing module 402, configured to establish a mapping relationship, where the mapping relationship includes a case number of each test case, one or more of a data insertion statement, a data deletion statement, and a data update statement corresponding to each test case, and a mapping relationship between one or more data rollback operation statements corresponding to the data insertion statement, the data deletion statement, and the data update statement corresponding to each test case;

a data recovery preparation module 403, configured to determine, based on the case number of the specified test case and the mapping relationship, one or more of a data insertion statement, a data deletion statement, and a data update statement corresponding to the corresponding target test case, or one or more data rollback operation statements corresponding to each of the data insertion statement, the data deletion statement, and the data update statement corresponding to the target test case; wherein the specified test case is any one of the plurality of test cases;

the data recovery executing module 404 is configured to execute one or more of the data inserting statement, the data deleting statement, and the data updating statement corresponding to the target test case, or execute one or more data rollback operation statements corresponding to the data inserting statement, the data deleting statement, and the data updating statement of the target test case, so as to implement data recovery.

In one embodiment, the generating, by the interception construction module, a data rollback operation statement corresponding to each of the data insertion statement, the data deletion statement, and the data update statement includes:

In one embodiment, the intercepting construction module intercepts one or more of a data insertion statement, a data deletion statement and a data update statement in a test request sent by an application system to a database when a plurality of test cases are executed in sequence, and the intercepting construction module comprises:

In one embodiment, the system further comprises:

the configuration module is used for modifying a first configuration file of the database before one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent to the database by the interception construction module when the application system executes a plurality of test cases in sequence so as to start a binary log function; wherein the first configuration file is used for opening or closing a binary log function of the database;

the establishing module is used for establishing a first data table in the database;

and the inserting module is used for inserting the use case number of the test case corresponding to each corresponding test request into the first data table before the database executes one or more of the data inserting statement, the data deleting statement and the data updating statement in each test request after the intercepting and constructing module intercepts one or more of the data inserting statement, the data deleting statement and the data updating statement in each test request, so that the inserting operation of the first data table is automatically recorded in a binary log of the database.

In an embodiment, the configuration module is further configured to modify a second configuration file of the database to open a database global transaction ID before one or more of a data insertion statement, a data deletion statement, and a data update statement in a test request sent by the interception construction module to the database when the application system executes the plurality of test cases in sequence; wherein the second configuration file is used for starting or closing a global transaction ID function of the database;

the establishing module is further used for establishing a second data table in the database;

the interception construction module generates data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data updating statement, and specifically comprises the following steps: analyzing the binary log to obtain at least one executable SQL statement and generating a data rollback SQL statement corresponding to each SQL statement; the at least one SQL statement is one or more related SQL statements in a data insertion statement, a data deletion statement and a data updating statement;

the mapping relationship establishing module is used for: analyzing the binary log to obtain a global transaction ID of at least one recorded transaction, and determining one or more SQL statements submitted by the transaction indicated by each global transaction ID based on the global transaction ID of the at least one transaction;

judging whether one or more SQL statements submitted by the transaction indicated by each global transaction ID have the insertion operation in the first data table, if so, determining the corresponding global transaction ID as a target global transaction ID, and acquiring a case number of a test case corresponding to the transaction indicated by the target global transaction ID; establishing a first mapping relation between the use case number of the test case corresponding to the transaction indicated by the target global transaction ID and the target global transaction ID, simultaneously establishing a second mapping relation between the target global transaction ID and one or more SQL statements submitted by the transaction indicated by the target global transaction ID and data rollback SQL statements corresponding to the one or more SQL statements, establishing the mapping relation based on the first mapping relation and the second mapping relation, and writing the mapping relation into the second data table.

In one embodiment, the system further comprises: the acquisition module is used for acquiring the binary logs of the database from the main node of the database based on a slave node simulation device in a preset listener before the mapping relation establishment module or the interception construction module analyzes the binary logs;

the intercepting construction module analyzes the binary log to obtain at least one executable SQL statement, and generates a data rollback SQL statement corresponding to each SQL statement, including:

and calling the slave node simulation device to analyze the binary log to obtain at least one executable SQL statement, and generating a data rollback SQL statement corresponding to each SQL statement.

In one embodiment, the mapping relationship establishing module establishes a mapping relationship, including: and establishing the mapping relation based on a mapping device in the listener.

With regard to the system in the above embodiment, the specific manner in which each module performs operations and the corresponding technical effects have been described in detail in the embodiment related to the method, and will not be described in detail herein.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units. The components shown as modules or units may or may not be physical units, i.e. may be located in one place or may also be distributed over a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the wood-disclosed scheme. One of ordinary skill in the art can understand and implement without inventive effort.

The disclosed embodiments also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the non-intrusive test data recovery method according to any of the above embodiments.

By way of example, and not limitation, such readable storage media can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

An embodiment of the present disclosure also provides an electronic device, including a processor and a memory for storing a computer program. Wherein the processor is configured to perform the non-intrusive test data recovery method of any of the above embodiments via execution of the computer program.

An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 5. The electronic device 600 shown in fig. 5 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 5, the electronic device 600 is in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 that connects the various system components (including the storage unit 620 and the processing unit 610), a display unit 640, and the like.

Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention as described in the above-mentioned method embodiment section of the present specification. For example, the processing unit 610 may perform the steps of the method as shown in fig. 1.

The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM) 6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.

The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 630 may be one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, or a network device, etc.) to perform the steps of the non-intrusive test data recovery method according to the above-mentioned embodiments of the present disclosure.

It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

The previous description is only for the purpose of describing particular embodiments of the present disclosure, so as to enable those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for non-intrusive test data recovery, comprising:

determining one or more of a data insertion statement, a data deletion statement and a data updating statement corresponding to a corresponding target test case based on a case number of a specified test case and the mapping relation, or determining a data rollback operation statement corresponding to one or more of the data insertion statement, the data deletion statement and the data updating statement corresponding to the target test case; wherein the specified test case is any one of the plurality of test cases;

2. The method according to claim 1, wherein the generating of the data rollback operation statement corresponding to each of the data insertion statement, the data deletion statement, and the data update statement comprises:

3. The method according to claim 1 or 2, wherein the intercepting application system sends one or more of a data insertion statement, a data deletion statement and a data update statement in a test request to the database when executing a plurality of test cases in sequence, and comprises:

intercepting one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent to the database by each test case based on an agent program; wherein, the agent is implemented based on the Java language and introduced into the application system in the form of a JAR package.

4. The method according to claim 1 or 2, wherein the intercepting application system sends one or more of a data insertion statement, a data deletion statement and a data update statement to a test request in the database when a plurality of test cases are executed in sequence, and the method comprises:

establishing a first data table in the database;

5. The method of claim 4, wherein the intercepting application system sends one or more of a data insert statement, a data delete statement, and a data update statement to the database in the test request when executing the plurality of test cases in sequence, and the method comprises:

establishing a second data table in the database;

the establishing of the mapping relation comprises the following steps:

judging whether one or more SQL statements submitted by the transaction indicated by each global transaction ID have the insertion operation in the first data table, if so, determining the corresponding global transaction ID as a target global transaction ID, and acquiring a case number of a test case corresponding to the transaction indicated by the target global transaction ID;

establishing a first mapping relation between the use case number of the test case corresponding to the transaction indicated by the target global transaction ID and the target global transaction ID, establishing a second mapping relation between one or more SQL statements submitted by the transaction indicated by the target global transaction ID and data rollback SQL statements corresponding to the one or more SQL statements, establishing the mapping relation based on the first mapping relation and the second mapping relation, and writing the mapping relation into the second data table.

6. The method of claim 5, wherein prior to parsing the binary log, the method further comprises:

7. The method of claim 6, wherein the establishing a mapping relationship comprises: and establishing the mapping relation based on a mapping device in the listener.

8. A non-intrusive test data recovery system, comprising:

the system comprises an interception construction module, a database and a plurality of application systems, wherein the interception construction module is used for intercepting one or more of a data insertion statement, a data deletion statement and a data updating statement in a test request sent to the database by the application systems when the application systems execute a plurality of test cases in sequence, and correspondingly generating data rollback operation statements corresponding to the data insertion statement, the data deletion statement and the data updating statement; each test request at least carries a case number of a corresponding test case and one or more of a data insertion statement, a data deletion statement and a data updating statement, and the case number of each test case is used for uniquely identifying the test case and indicating the execution sequence of the test case;

the mapping relation establishing module is used for establishing a mapping relation, wherein the mapping relation comprises a case number of each test case, one or more of a data inserting statement, a data deleting statement and a data updating statement corresponding to each test case, and the mapping relation among one or more data rollback operation statements respectively corresponding to the data inserting statement, the data deleting statement and the data updating statement corresponding to each test case;

9. A computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the non-invasive test data recovery method of any of claims 1~7.

10. An electronic device, comprising:

a processor; and

a memory for storing a computer program;

wherein the processor is configured to perform the non-intrusive test data recovery method of any of claims 1~7 via execution of the computer program.