CN111382198B - Data recovery method, device, equipment and storage medium - Google Patents
Data recovery method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111382198B CN111382198B CN201811629215.4A CN201811629215A CN111382198B CN 111382198 B CN111382198 B CN 111382198B CN 201811629215 A CN201811629215 A CN 201811629215A CN 111382198 B CN111382198 B CN 111382198B
- Authority
- CN
- China
- Prior art keywords
- data
- value
- field
- predetermined
- tables
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
Abstract
The invention discloses a data recovery method, a device, equipment and a storage medium, wherein the method comprises the following steps: determining at least one table associated with a predetermined field in a predetermined table according to the predetermined field; forming an association relation template between tables according to the association relation between the preset table and the at least one table; collecting and backing up table data associated with each value in at least one value of the predetermined field according to the association relation template; and acquiring table data associated with a target value in the at least one value from the backed-up table data so as to perform data restoration. According to the embodiment of the invention, the batch synchronization efficiency of the relational database can be improved, the cost can be reduced, and the automatic regression test can be completed in a fixed time and a fixed quantity without manual intervention.
Description
Technical Field
The present invention relates to the field of computers, and in particular, to a data recovery method, apparatus, device, and storage medium.
Background
It is well known that databases are an integral part of indelibility. During software development, debugging, testing, programmers need to constantly test the current application to ensure the correctness, accuracy, and robustness of the software. Most tests are to verify whether an application can perform the intended add-delete-modify operations on specific data within the database.
Some of the test data contents belong to non-renewable resources, for example, after a test of adding operation is performed on one piece of data in the database, the target expected data already exists, which is often limited by a primary key, a unique index or a business rule, and the current data contents cannot be used for performing new addition operation. Unless the current data content is deleted by manual operation, or a piece of data content meeting the test requirement is additionally searched, or the data in the database is restored. At present, there are two ways to realize data recovery:
1. relational database batch recovery
The main stream relational database mainly completes batch application data restoration in a mode of batch synchronization or data snapshot of backup data, and the data sources are backup databases and logs. The method aims to mainly deal with database information security and reserve customer data to the greatest extent in sudden faults such as power failure.
For example, the mainstream Oracle and MySQL databases support large-batch data synchronization through self application functions and auxiliary internal or third party tools such as SQLLOAD, goodenGate; oracle itself can reply data occurring in minutes or even hours in an emergency, depending on the underlying capabilities.
2. Non-relational database batch recovery
The main flow of non-relational databases, including Hadoop, hbase, monggoDB and other databases, mainly stores data in a 1-standby N mode through an internal master-standby and emergency mechanism, but when one data storage node fails to enable the data to be unavailable or the data is lost, the management node can automatically start and execute a security policy, synchronize the data of other data nodes and continuously maintain the data persistence state of 1-standby N of one data.
The data source of the non-relational database comes from the storage and adopts a high availability mechanism, and the purpose of the data source is mainly to deal with the information security of the database and to reserve the customer data to the greatest extent in sudden faults such as fault nodes, downtime and the like.
3. Manual maintenance of small data synchronization
Whether the relational database or the non-relational database, the small amount of target data can be added, deleted and changed by manually compiling a data operation script, such as a structured query language (Structured Query Language, SQL) script. And the reduction of a small amount of target data is basically realized by reversely writing a rollback script, backing up the local online script and the like.
The artificial script data maintenance has strong operability, wide application range and extremely strong flexibility, and is generally used for daily data maintenance. The system also comprises application data restoration functions in the application development and test processes.
Disadvantages of the prior art:
1. the batch data recovery cost is higher, and the rapid test requirement at any time and any need cannot be met: the problem that the content of the test data is unavailable due to the fact that the target data is changed can be solved by regularly recovering the data of the test database and the research database, but one test scene can relate to parallel recovery of a plurality of databases and is high in cost for large-scale industry application; meanwhile, the recovered data content is still unavailable after the test is used, and a target is required to be searched or the next batch recovery is required to be waited.
2. The manual maintenance efficiency is low, and resources are difficult to share: restoring test data by means of artificial scripts is a common practice in the software development process. And a programmer or a tester writes a rollback script temporarily according to the application and service scene cognition, so that the next debugging and use are convenient. The temporarily composed script is often only specific to one or a few specific scenarios, once the test is completed, the script is often invalidated and rewritten next time.
3. Complexity problem of industry application data: whether manually composing a reply script or batch database data recovery, the complexity of industry application data is faced. Such as: time problem, yesterday, test data available in the last month, cannot be used today or in the present month because of restrictions of business rules, specifically, as follows: when the discount is transacted in month, the data of the last month is not available whether the data is recovered manually or the database is recovered. At this point, the order date needs to be manually modified (UPDATE) to be the "date of manufacture" phenomenon common to the software industry, which is the day of the month.
Disclosure of Invention
The embodiment of the invention provides a data recovery method, a device, equipment and a storage medium, which can solve the problems of low efficiency and high cost of batch synchronization of a relational database, and can complete automatic regression testing in a timed and quantitative manner without depending on manual intervention.
In a first aspect, an embodiment of the present invention provides a data reduction method, where the method includes:
determining at least one table associated with a predetermined field in a predetermined table according to the predetermined field;
forming an association relation template between tables according to the association relation between the preset table and the at least one table;
collecting and backing up table data associated with each value in at least one value of the predetermined field according to the association relation template;
and acquiring table data associated with a target value in the at least one value from the backed-up table data so as to perform data restoration.
In a second aspect, an embodiment of the present invention provides a data recovery apparatus, including:
the association table determining module is used for determining at least one table associated with the preset field according to the preset field in the preset table;
the template forming module is used for forming an association relation template between the tables according to the association relation between the preset table and the at least one table;
the data acquisition module is used for acquiring and backing up table data associated with each value in at least one value of the preset field according to the association relation template;
and the data recovery module is used for acquiring the table data associated with the target value in the at least one value from the backup table data so as to recover the data.
In a third aspect, an embodiment of the present invention provides a data recovery apparatus, including: a processor and a memory storing computer program instructions;
the processor implements the data reduction method when executing the computer program instructions.
In a fourth aspect, embodiments of the present invention provide a computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement the data reduction method.
The data reduction method, the device, the equipment and the storage medium provided by the embodiment of the invention can be used for carrying out data reduction through the table data related to one or more target values from a specific scene, have high data reduction efficiency, can complete the data reduction of one test case in a short time, can be reused after the test, and solve the problems of low batch synchronization efficiency and high cost of the relational database. Moreover, the association relation template is introduced to solve the complexity problem of industry data restoration, fields of each table can be customized according to the needs of specific scenes, automatic restoration of data is realized, and reusability is greatly improved. Furthermore, the method can support large-scale massive background regression testing, and automatic regression testing can be completed in a fixed time and in a fixed quantity without manual intervention after test scenes and data reduction are established.
Drawings
In order to more clearly illustrate the technical solution of the embodiments of the present invention, the drawings that are needed to be used in the embodiments of the present invention will be briefly described, and it is possible for a person skilled in the art to obtain other drawings according to these drawings without inventive effort.
FIG. 1 is a schematic flow chart of a data recovery method according to an embodiment of the present invention;
FIG. 2 shows a schematic diagram of fields in a data table according to one embodiment of the invention;
FIG. 3 is a diagram showing the association between fields according to one embodiment of the present invention;
FIG. 4 is a diagram showing the relationship between tables according to one embodiment of the present invention;
FIG. 5 is a schematic diagram of fields in a data table according to another embodiment of the present invention;
FIG. 6 is a diagram showing association between fields according to another embodiment of the present invention;
FIG. 7 is a diagram showing the relationship between tables according to another embodiment of the present invention;
FIG. 8 is a simplified diagram of table associations according to one embodiment of the invention;
FIG. 9 is a schematic flow chart of a data recovery method according to another embodiment of the present invention;
FIG. 10 is a flow diagram of a configuration association template according to one embodiment of the present invention;
FIG. 11 shows a schematic flow chart diagram of acquisition table data of one embodiment of the invention;
FIG. 12 shows a flow diagram of a template restore process of one embodiment of the invention;
FIG. 13 is a schematic diagram of a data recovery device according to an embodiment of the present invention;
fig. 14 shows a schematic hardware structure of a data recovery device according to an embodiment of the present invention.
Detailed Description
Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings and the detailed embodiments. It should be understood that the specific embodiments described herein are merely configured to illustrate the invention and are not configured to limit the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the invention by showing examples of the invention.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
In order to solve the problems in the prior art, the embodiment of the invention provides a data recovery method, a device, equipment and a computer storage medium. The following first describes a data recovery method provided by an embodiment of the present invention.
Fig. 1 is a schematic flow chart of a data recovery method according to an embodiment of the present invention. As shown in fig. 1, the method includes:
s101, determining at least one table associated with the predetermined field according to the predetermined field in the predetermined table.
S102, forming an association relation template between the tables according to the association relation between the preset table and at least one table.
S103, acquiring and backing up table data associated with each value in at least one value of the predetermined field according to the association relation template.
S104, acquiring the table data associated with the target value in the at least one value from the backed-up table data so as to restore the data.
For example, after S103, a test is performed, data is changed due to the test or other reasons, and S104 may be performed to perform data restoration.
It should be noted that, the embodiment of the invention realizes automatic restoration of local data based on the field association relationship between the data tables in the relational database. When a field value of a data table is known, all associated object contents related to a transaction in the data table affected by the previous transaction in the whole database can be automatically restored according to the field value. Such as: in the mobile communication industry, after a product change test is performed on a mobile phone number, the content of a data line in a series of tables related to the mobile phone number in the current database environment is restored, so that secondary test or other test work can be conveniently performed.
In the embodiment of the invention, firstly, from a specific scene, data reduction is carried out through table data associated with one or more target values, the data reduction efficiency is high, the data reduction of one test case can be completed in a short time (about 2 minutes), the test case can be reused, and the problems of low batch synchronization efficiency and high cost of the relational database are solved. And secondly, the association relation template is introduced to solve the complexity problem of industry data restoration, the fields of each table can be customized according to the requirements of specific scenes, the automatic restoration of the data is realized, the reusability is greatly improved, and the complexity problem of industry application data is solved. Finally, the data can be multiplexed and customized to special field characteristics, and can support large-scale massive background regression testing, and automatic regression testing can be completed in a timed and quantitative manner without manual intervention after test scenes and data restoration are established.
In one embodiment of the present invention, S101 includes:
taking a predetermined field as a starting field; searching a field associated with the initial field, and taking a table in which the field is located as a table to be processed; returning each field in the table to be processed as a new initial field to the step of searching until a first preset condition is met; and taking all the obtained tables to be processed as at least one table.
Wherein the first predetermined condition includes: the fields associated with the initial fields or the obtained pending tables cannot be found to meet the test scenario requirements.
In the actual production and application work of the database, the design and development are not to completely use the main external key of the database to maintain the data consistency based on factors such as efficiency, service complexity and the like, but to maintain the service data consistency in service logic codes through upper WEB or middleware application. FIG. 2 shows a schematic diagram of fields in a data table according to one embodiment of the invention. As shown in fig. 2, the relationships between the fields are distinguished by the outline shape of each field, but it is of course also possible to distinguish the relationships between the fields by the colors of each field in the table. In fig. 2, field 1 in data table a and field 2 in data table B are both square outer frames, i.e., field 1 in data table a and field 2 in data table B have an association relationship. Similarly, field 1 in data table B and field 1 in data table C have an association relationship, and field 3 in data table C and field 1 in data table D have an association relationship.
From the field association relationships in the tables in fig. 2, a schematic diagram of the field association relationships shown in fig. 3 can be obtained. As shown in fig. 3, A1 and B2 have an association relationship, B1 and C1 have an association relationship, C3 and D1 have an association relationship, A1 represents field 1 in the data table a, B2 represents field 2 in the data table B, and the meanings of B1, C3, and D1 can be similarly used.
Fig. 2 enumerates the association of 4 data tables, i.e. when the instance value of A1 (e.g. the user PHONE number phone_no) is obtained, one or a group of instance values of B2 can be obtained immediately by SQL statements, one or a group of one-to-one, one-to-many or many-to-many relationships depending on A1 and B2.
After B2 is obtained based on A1, B1 can be directly obtained in a data B table where B2 is located, and then one or more groups of C1 are obtained. Similarly, a list of all D1 fields may be derived from a list of all derived C1 fields.
FIG. 4 is a schematic diagram showing the relationship between tables according to one embodiment of the present invention, wherein the A-table data row is acquired in dependence on A1 and the B-table data row is acquired in dependence on A1, both of which are direct acquisitions; the C table data line depends on B2 data acquisition obtained by A1, and the D table data line depends on C3 data acquisition in C1 data content obtained by B1, and the two data acquisition are indirect acquisition. In fig. 4, the solid line represents a direct acquisition relationship, and the broken line represents an indirect acquisition relationship.
In practical application, the data model is designed according to the specific application requirement and most of the increment is evolved, so that the association relation result of the transaction data table in the general industry has a graph structure. The embodiment of the invention converts the graph association relationship into a binary tree association relationship.
Fig. 5 shows a schematic diagram of fields in a data table according to another embodiment of the invention. As shown in fig. 5, on the basis of four tables of user basic information (data table a), user subscription information (data table B), user subscription attribute information (data table C), and user attribute configuration information (data table D), a service operation rule record table X is newly added according to service requirements, and the association relationship between the fields is shown in fig. 6. In fig. 6, the field association relationship is complex and redundant, and in the case that a certain field instance value of the data A1 is known, there are various ways to acquire the data row content of the B, C, D table, as shown in fig. 7, if it is known that A1 needs to acquire B table data, it is possible to: and if the B table is acquired in three modes, namely A1-B2, A1-X1-B1, A1-X1-C3-B1 and the like, repeated operation is caused when the acquired table data are redundant and restored, so that the research and development landing cost is increased, and the processing efficiency is reduced.
Based on the above problems, in one embodiment of the present invention, the method further includes:
after a plurality of tables to be processed are obtained, judging whether repeated tables exist in the tables to be processed; if there is a duplicate table, one of the duplicate tables is maintained and the other of the duplicate tables is deleted.
The association relation template is a binary tree-shaped association relation template.
In the embodiment of the invention, for repeated tables, only one table is reserved, and other tables are deleted, for example, table a and table B are repeated, and then one table in table a and table B is reserved, and the other table is deleted. The repeated table is avoided, so that the problems of multiple association modes, redundant storage and repeated restoration exist, and the association complexity is reduced, and the recovery efficiency is improved. As shown in fig. 8, duplicate tables are removed, greatly simplifying the association relationship between tables.
In one embodiment of the present invention, S103 includes:
for each of the at least one value, performing a predetermined step to obtain table data associated with each of the at least one value.
Wherein the predetermined steps include:
taking the current value to be processed in the at least one value as a starting value; sequentially collecting at least one table to be collected associated with the initial value according to the order of the tables in the association relation template; returning to execute at least one table to be acquired associated with the new starting value by taking the field value in each table to be acquired as the new starting value until a second preset condition is met; and taking the data in all the acquired tables as table data associated with the current value to be processed.
Wherein the second predetermined condition includes: there is no table to be acquired associated with the start value within a predetermined acquisition range.
As an example, all the data row contents related to the A, B, C and D tables can be obtained based on the field value of A1, and the data row content record can be used for subsequent data restoration operation after being backed up.
For example: when table data associated with a specific mobile phone number is acquired, the mobile phone number is taken as a starting value, and a user ordering information table associated with the mobile phone number is acquired according to the sequence of the table in the association relation template; secondly, taking a field value in the user ordering information table as a new starting value, and collecting a user ordering attribute information table associated with the new starting value; again, the field value in the user subscription attribute information table is taken as a new starting value, and the user attribute configuration information table associated with the new starting value is collected. Therefore, the user order information table, the user order attribute information table, and the data in the user attribute configuration information table are sequentially collected according to the order of the table in the association relationship template.
In one embodiment of the present invention, S104 includes:
at the time of data restoration, table data within a predetermined restoration range is replaced with table data associated with a target value.
In one embodiment of the present invention, S104 includes:
the data of the table in the predetermined restoration range is deleted by the deletion instruction, and the table data associated with the target value is inserted by the insertion instruction.
Fig. 9 is a schematic flow chart of a data recovery method according to another embodiment of the present invention. As shown in fig. 9, the method includes:
s201, configuring an association relation template based on the association relation of the fields.
Wherein S201 includes:
s2011, according to business test and automatic test requirements, a preset table and a preset field are selected and recorded. Such as: in the mobile industry, the data list and the data row which are directly and indirectly related are started by the mobile phone number under the user basic list, and the data record and the restoration can meet the service test requirement of more than 95 percent.
S2012, the mapping relation is combed, the preset field is used as a starting field, and all tables directly related to the starting field are searched in a database and recorded (or backed up). The recorded tables are culled at recording time. The priority order of the next data acquisition is defined at recording time.
And S2013, taking the fields in the data table obtained in the step S2012 as new initial fields, and repeating the action of the lookup table in the step S2012 until all the fields in the data table obtained in the step S2012 are searched. The recorded tables are culled at recording time.
And S2014, taking the field in the data table obtained in the step S2013 as a new initial field, and repeating the action of the lookup table in the step S2012 until the data result obtained in the step S2013 meets the test scene requirement. And rejecting the recorded data table when recording.
S2015, persisting the result of S2014, and for the predetermined field of the predetermined table, there is no need to execute the operation of S201 again.
So far, based on the configuration of the association relation template of the preset field, delivering a relation data instance result between the service test dimension data table of the binary tree result and the field, namely the association relation template. The association relation template is independent of specific values of a preset table and preset fields, and any field value which is not empty can be used for directly or indirectly positioning data rows in a full data table required by a data restoration scene by using the association relation template.
Fig. 10 shows a flow diagram of configuring an association template according to an embodiment of the present invention. As shown in FIG. 10, the template configuration application is responsible for storing the business association relationship in the target database after artificial combing, and entering the configuration code table of the management database, wherein the configuration is mainly pseudo SQL, that is, variables which can be dynamically acquired only when the acquisition execution or the restoration execution exists in the assignment part of the SQL statement, such as operation time, operation flow and the like.
S202, collecting data in a table based on at least one field value and the association relation template, and backing up the collected data.
The delivery of S202 is specific data for the data rows in the full data table required for the data recovery scenario, and is used to collect the data record before the operation in the data recovery scenario. Different start tables and field values in the start tables collect different data.
Wherein S202 includes:
s2021: at least one valid field value of the stock target template and the predetermined field is entered into the system application. The target template is used for defining the range of data acquisition and data restoration, and the effective field value is the data starting point of a specific restoration operation.
S2022: the system automatically takes the field value as a starting value according to the effective field value, and collects the whole content of the data row of the table associated with the starting value. And sequentially collecting the data of each table associated with the initial value according to the sequence of the tables in the association relation template, and backing up the data.
S2023: and repeating the operation in the step S2021 with the field value in the table obtained in the step S2022 as a new starting value until the data content obtained in the step S2022 is complete.
S2024: and repeating the step S2022 with the data result obtained in the step S2023 as a new starting value until the configuration relations in the association relation template in the step S2023 are all processed once.
S2025: the result of S2024 is stored persistently, and the operation of S202 need not be performed every time the test scene data is restored later.
So far, based on at least one field value and the association relation template, data in the table are collected, backup of the collected data is completed, and a result set recorded before the business test operation of the binary tree result is delivered. The result set is independent of any test scene, and the tests in all operation table ranges can be reused. Such as: in the mobile communication industry, after the operation of S202 is completed by the mobile phone number under the user basic list, more than about 95% of service test requirements can be met.
FIG. 11 shows a schematic flow chart of acquisition table data according to one embodiment of the invention. As shown in FIG. 11, the template collection application is mainly composed of a general variable pool and a pseudo SQL execution block in the interior, the general variable pool is used as a general access area, the encountered pseudo SQL variables K-V are sequentially recorded in a key value pair mode, and the current values of the variables in the pseudo SQL are dynamically obtained according to constants, environment variables or SQL sentences of the configuration of the data model. This loops until all pseudo SQL values required during all acquisitions are acquired. After the collection of the pseudo SQL values is completed, the model collection application sequentially adds the Deletion (DELETE) pseudo SQL and the Insertion (INSERT) pseudo SQL into a table in a mode of processing the data instance by a template according to the management data model configuration.
S203, based on the target field value, the association relation template and the backed-up data, the database data restoring operation is executed.
S203 delivers the pre-test data preparation phase for all data contents of the data restoration of the test. This step is generated in the relational database and the test work can be deployed based on the data backed up at S202. This step may be repeated any number of times, such as: after executing S203, one mobile phone number may be used to test repeatedly, so as to verify the correctness of the system application in the test environment before the online, or the correctness of the system in the production environment after the serious online.
Wherein S203 includes:
s2031, an inventory target template and a valid target value are input to the system application. The template is used to define the data collection and reduction range, the target value is the data start point of a specific reduction operation, the target value is a value in at least one valid field value in S2021, and the number of target values may be one or more.
S2032, the system automatically extracts table data associated with the target value from the data backed up in S202, replaces KEY variables in the pseudo SQL for deleting the data one by one, and executes the data after generating the deleted SQL.
The purpose of deleting SQL is to delete the historical data generated by the last test or other reason.
S2033, extracting table data associated with the target value from the data backed up in S202, replacing KEY variables in the pseudo SQL of data restoration one by one, generating restoration SQL, and executing.
And S2034, after all the execution of the step S2033 is completed and all the execution is correct, submitting the database, and feeding back success information to the upstream of the system. After all execution is completed and an error occurs in S2033, the database rolls back and failure information is fed back to the system upstream for locating vulnerabilities (BUGs).
So far, based on the target value, the association relation template and the backed-up data, the database data after the operation is restored is completed, and the relational database content restoration required by one test is delivered. S203 is repeatedly called, executed, and delivered at each specific test.
FIG. 12 shows a flow diagram of a template restore process of one embodiment of the invention. As shown in FIG. 12, the template recovery processing application is mainly composed of a general variable pool and a pseudo SQL execution block, the general variable pool is used as a general access area, the encountered pseudo SQL variables K-V are sequentially recorded in a key value pair mode, and the current values of the variables in the pseudo SQL are dynamically obtained according to constants, environment variables or SQL sentences of the configuration of a data model. This loops until all pseudo SQL values required during the restore process are obtained.
According to the embodiment of the invention, firstly, from the business logic relationship, the data required by the test can be selectively and automatically restored after the template design and the record storage, the problem of test environment maintenance under the relational database is solved, and particularly, the customer relationship management (Customer Relationship Management, CRM) system in the support system of the mobile communication industry can greatly improve the daily development test efficiency.
Secondly, based on the business logic relation of the data tables and the fields in the relational database, the application design, the flow design and the model design of the data restoration scene are realized, the problem of low operation efficiency of data restoration is solved, the complex problem of data restoration of different data tables in different test scenes is solved, the test efficiency is improved, and the method can be used for automatic regression test of an automatic and large-scale system.
The embodiment of the invention at least comprises the following four advantages:
1. the method solves the problems of low batch synchronization efficiency and high cost of the relational database, and the embodiment of the invention starts from single-database local data in a specific test scene, has high data reduction efficiency, can complete the full data reduction of one test case within 2 minutes, and can be reused after the test.
2. The embodiment of the invention adopts an independent data maintenance system to automatically generate and deliver the DELETE and INSERT scripts for data reduction in the test cases, the same test number can be used as required, and no abnormality exists as long as the test time (within 2 minutes) is different.
3. The method and the device solve the complexity problem of the industry application data, and solve the complexity problem of the industry application data through a multi-template, external variables and pseudo SQL modes, special fields of each data table can be customized according to specific scene requirements, and data generation is automated, so that reusability is greatly improved.
4. Based on the special field characteristics that the proposed data can be multiplexed and customized, the embodiment of the invention can support large-scale massive background regression testing, and after a test scene and data reduction are established, automatic regression testing can be completed regularly and quantitatively without relying on manual intervention, which cannot be provided by the traditional scheme.
Fig. 13 is a schematic diagram showing the structure of a data restoring apparatus according to an embodiment of the present invention. As shown in fig. 13, the apparatus 300 includes:
the association table determining module 301 is configured to determine at least one table associated with a predetermined field according to the predetermined field in the predetermined table.
The template forming module 302 is configured to form an association relationship template between tables according to an association relationship between a predetermined table and at least one table.
The data collection and backup module 303 is configured to collect and backup table data associated with each value of at least one value of the predetermined field according to the association relation template.
The data restoring module 304 is configured to obtain, from the backed up table data, table data associated with a target value of the at least one value, so as to restore the data.
In one embodiment of the present invention, the association table determination module 301 includes:
the first initial field determining module is used for taking a preset field as an initial field;
and the table searching module is used for searching a field associated with the initial field and taking a table in which the field is positioned as a table to be processed.
And the second initial field determining module is used for taking each field in the table to be processed as a new initial field.
The table searching module is further configured to repeat the step of performing the searching until a first predetermined condition is satisfied.
And the first table determining module is used for taking all the obtained tables to be processed as at least one table.
In one embodiment of the present invention, the first predetermined condition includes: the fields associated with the initial fields or the obtained pending tables cannot be found to meet the test scenario requirements.
In one embodiment of the present invention, the apparatus 300 further comprises:
the judging module is used for judging whether repeated tables exist in the multiple tables to be processed after the multiple tables to be processed are obtained;
and the deduplication module is used for reserving one table in the duplicate tables and deleting other tables in the duplicate tables when the duplicate tables exist.
In one embodiment of the invention, the association template is a binary tree-like association template.
In one embodiment of the invention, the data collection and backup module 303 comprises:
a data acquisition module for performing a predetermined step on each of the at least one value to obtain table data associated with each of the at least one value.
Wherein, the data acquisition module includes:
and the first starting value determining module is used for taking the current value to be processed in the at least one value as a starting value.
And the table acquisition module is used for sequentially acquiring at least one table to be acquired which is associated with the initial value according to the sequence of the tables in the association relation template.
And the second starting value determining module is used for taking the field value in each table to be acquired as a new starting value.
The table acquisition module is further configured to repeatedly execute at least one table to be acquired associated with the new start value until a second predetermined condition is satisfied.
And the second table determining module is used for taking the data in all the acquired tables as table data associated with the current value to be processed.
In one embodiment of the invention, the second predetermined condition comprises: there is no table to be acquired associated with the start value within a predetermined acquisition range.
In one embodiment of the invention, the data reduction module 304 includes:
and the data replacing module is used for replacing the table data in the preset restoring range with the table data associated with the target value.
In one embodiment of the present invention, the data replacement module includes:
and the data deleting module is used for deleting the data of the table in the preset restoring range through a deleting instruction.
And the data insertion module is used for inserting the table data associated with the target value through the insertion instruction.
In one embodiment of the invention, the data collection and backup module is configured to collect table data associated with each of the at least one value of the predetermined field in the relational database.
And the data restoration module is used for restoring the data in the relational database according to the table data associated with the target value.
Fig. 14 shows a schematic hardware structure of a data recovery device according to an embodiment of the present invention.
The data reduction device may include a processor 401 and a memory 402 in which computer program instructions are stored.
In particular, the processor 401 described above may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured as one or more integrated circuits implementing embodiments of the present invention.
Memory 402 may include mass storage for data or instructions. By way of example, and not limitation, memory 402 may comprise a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, magnetic tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the foregoing. Memory 402 may include removable or non-removable (or fixed) media, where appropriate. Memory 402 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 402 is a non-volatile solid state memory. In a particular embodiment, the memory 402 includes Read Only Memory (ROM). The ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these, where appropriate.
The processor 401 implements any of the data recovery methods of the above embodiments by reading and executing computer program instructions stored in the memory 402.
In one example, the data reduction device may also include a communication interface 403 and a bus 410. As shown in fig. 14, the processor 401, the memory 402, and the communication interface 403 are connected to each other by a bus 410 and perform communication with each other.
The communication interface 403 is mainly used to implement communication between each module, device, unit and/or apparatus in the embodiment of the present invention.
Bus 410 includes hardware, software, or both, coupling the components of the data reduction device to one another. By way of example, and not limitation, the buses may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a micro channel architecture (MCa) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 410 may include one or more buses, where appropriate. Although embodiments of the invention have been described and illustrated with respect to a particular bus, the invention contemplates any suitable bus or interconnect.
The data restoring apparatus may perform the data restoring method in the embodiment of the present invention, thereby implementing the data restoring method and apparatus described in connection with fig. 1 and 13.
In addition, in combination with the data recovery method in the above embodiment, the embodiment of the present invention may be implemented by providing a computer storage medium. The computer storage medium has stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the data reduction methods of the above embodiments.
It should be understood that the invention is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present invention.
The functional blocks shown in the above-described structural block diagrams may be implemented in hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transfer information. Examples of machine-readable media include electronic circuitry, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and the like. The code segments may be downloaded via computer networks such as the internet, intranets, etc.
It should also be noted that the exemplary embodiments mentioned in this disclosure describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, or may be performed in a different order from the order in the embodiments, or several steps may be performed simultaneously.
In the foregoing, only the specific embodiments of the present invention are described, and it will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein. It should be understood that the scope of the present invention is not limited thereto, and any equivalent modifications or substitutions can be easily made by those skilled in the art within the technical scope of the present invention, and they should be included in the scope of the present invention.
Claims (13)
1. A method of data recovery, comprising:
determining at least one table associated with a predetermined field in a predetermined table according to the predetermined field;
forming an association relation template between tables according to the association relation between the preset table and the at least one table;
collecting and backing up table data associated with each value in at least one value of the predetermined field according to the association relation template;
and acquiring table data associated with a target value in the at least one value from the backed-up table data so as to perform data restoration.
2. The method of claim 1, wherein determining at least one table associated with a predetermined field in the predetermined table based on the predetermined field comprises:
taking the preset field as a starting field;
searching a field associated with the initial field, and taking a table in which the field is located as a table to be processed;
returning each field in the table to be processed as a new initial field to the step of executing the search until a first preset condition is met;
and taking all the obtained tables to be processed as at least one table.
3. The method of claim 2, wherein the first predetermined condition comprises: and searching for a field associated with the initial field or obtaining the to-be-processed table which meets the test scene requirement.
4. The method as recited in claim 2, further comprising:
after a plurality of tables to be processed are obtained, judging whether repeated tables exist in the tables to be processed;
if the repeated table exists, one table in the repeated table is reserved, and other tables in the repeated table are deleted.
5. The method of claim 1, wherein the association template is a binary tree-like association template.
6. The method of claim 1, wherein collecting table data associated with each of the at least one value of the predetermined field according to the association template comprises:
performing a predetermined step on each of the at least one value to obtain table data associated with each of the at least one value;
wherein the predetermined step includes:
taking the current value to be processed in the at least one value as a starting value;
sequentially collecting at least one table to be collected associated with the initial value according to the order of the tables in the association relation template;
returning to execute at least one table to be acquired associated with the new starting value by taking the field value in each table to be acquired as the new starting value until a second preset condition is met;
and taking all the obtained data in the tables to be collected as table data associated with the current values to be processed.
7. The method of claim 6, wherein the second predetermined condition comprises: there is no table to be acquired associated with the start value within a predetermined acquisition range.
8. The method of claim 1, wherein the step of determining the position of the probe comprises,
and replacing table data in a preset restoring range with table data associated with the target value when the data is restored.
9. The method of claim 8, wherein,
the data of the table within the predetermined restoration range is deleted by the deletion instruction, and the table data associated with the target value is inserted by the insertion instruction.
10. The method of claim 1, wherein the step of determining the position of the probe comprises,
collecting table data associated with each of the at least one value of the predetermined field in a relational database;
and, in addition, the processing unit,
and restoring the data in the relational database according to the table data associated with the target value.
11. A data recovery apparatus, comprising:
the association table determining module is used for determining at least one table associated with the preset field according to the preset field in the preset table;
the template forming module is used for forming an association relation template between the tables according to the association relation between the preset table and the at least one table;
the data acquisition module is used for acquiring and backing up table data associated with each value in at least one value of the preset field according to the association relation template;
and the data recovery module is used for acquiring the table data associated with the target value in the at least one value from the backup table data so as to recover the data.
12. A data reduction apparatus, the apparatus comprising: a processor and a memory storing computer program instructions;
the processor, when executing the computer program instructions, implements a data reduction method as claimed in any one of claims 1-10.
13. A computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement a data reduction method according to any of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811629215.4A CN111382198B (en) | 2018-12-28 | 2018-12-28 | Data recovery method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811629215.4A CN111382198B (en) | 2018-12-28 | 2018-12-28 | Data recovery method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111382198A CN111382198A (en) | 2020-07-07 |
| CN111382198B true CN111382198B (en) | 2023-09-19 |
Family
ID=71214753
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811629215.4A Active CN111382198B (en) | 2018-12-28 | 2018-12-28 | Data recovery method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111382198B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112699187B (en) * | 2020-12-29 | 2023-05-16 | 中国联合网络通信集团有限公司 | Associated data processing method, device, equipment, medium and product |
| CN113127359A (en) * | 2021-04-23 | 2021-07-16 | 中国工商银行股份有限公司 | Method and device for obtaining test data |
| CN113204566B (en) * | 2021-06-09 | 2024-01-26 | 中国银行股份有限公司 | Execution method and device of SQL script |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6651077B1 (en) * | 2000-09-27 | 2003-11-18 | Microsoft Corporation | Backup and restoration of data in an electronic database |
| WO2007137468A1 (en) * | 2006-05-26 | 2007-12-06 | Huawei Technologies Co., Ltd. | Method and system for creating relational model automatically |
| CN102096669A (en) * | 2009-12-14 | 2011-06-15 | 深圳速浪数字技术有限公司 | Data backup method and data backup device |
| CN106445727A (en) * | 2015-08-07 | 2017-02-22 | 中国移动通信集团重庆有限公司 | Data backup method and system, and data recovery method and system |
| CN107220251A (en) * | 2016-03-21 | 2017-09-29 | 阿里巴巴集团控股有限公司 | Generate the method and device of description information |
| US9959175B1 (en) * | 2015-06-30 | 2018-05-01 | Spanning Cloud Apps, LLC | Restoring deleted objects in a web application |
| CN108509485A (en) * | 2018-02-07 | 2018-09-07 | 深圳壹账通智能科技有限公司 | Preprocess method, device, computer equipment and the storage medium of data |
| CN108563535A (en) * | 2018-04-27 | 2018-09-21 | 四川巧夺天工信息安全智能设备有限公司 | A kind of restoration methods to the full library of MySQL database |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8117187B2 (en) * | 2005-10-28 | 2012-02-14 | Mediareif Moestl & Reif Kommunikations-Und Informationstechnologien Oeg | Method for controlling a relational database system |
| US20130232106A1 (en) * | 2012-03-01 | 2013-09-05 | Cover-All Technologies, Inc. | System and Method For Applying An Update To A Database |
-
2018
- 2018-12-28 CN CN201811629215.4A patent/CN111382198B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6651077B1 (en) * | 2000-09-27 | 2003-11-18 | Microsoft Corporation | Backup and restoration of data in an electronic database |
| WO2007137468A1 (en) * | 2006-05-26 | 2007-12-06 | Huawei Technologies Co., Ltd. | Method and system for creating relational model automatically |
| CN102096669A (en) * | 2009-12-14 | 2011-06-15 | 深圳速浪数字技术有限公司 | Data backup method and data backup device |
| US9959175B1 (en) * | 2015-06-30 | 2018-05-01 | Spanning Cloud Apps, LLC | Restoring deleted objects in a web application |
| CN106445727A (en) * | 2015-08-07 | 2017-02-22 | 中国移动通信集团重庆有限公司 | Data backup method and system, and data recovery method and system |
| CN107220251A (en) * | 2016-03-21 | 2017-09-29 | 阿里巴巴集团控股有限公司 | Generate the method and device of description information |
| CN108509485A (en) * | 2018-02-07 | 2018-09-07 | 深圳壹账通智能科技有限公司 | Preprocess method, device, computer equipment and the storage medium of data |
| CN108563535A (en) * | 2018-04-27 | 2018-09-21 | 四川巧夺天工信息安全智能设备有限公司 | A kind of restoration methods to the full library of MySQL database |
Non-Patent Citations (1)
| Title |
|---|
| 王伟平 ; 王子卿. Oracle用户SQL会话还原方法研究.《计算机工程与应用》.2008,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111382198A (en) | 2020-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111382198B (en) | Data recovery method, device, equipment and storage medium | |
| CN104252481A (en) | Dynamic check method and device for consistency of main and salve databases | |
| CN109063005B (en) | Data migration method and system, storage medium and electronic device | |
| CN107479881B (en) | Method for synchronizing difference codes, storage medium, electronic device and system | |
| CN109634846B (en) | ETL software testing method and device | |
| CN110209585A (en) | A kind of software test case intelligent training method, terminal and storage medium | |
| CN101673374A (en) | Bill processing method and device | |
| CN111125229A (en) | Data blood margin generation method and device and electronic equipment | |
| CN114564500A (en) | Method and system for implementing structured data storage and query in block chain system | |
| CN106155838A (en) | A kind of database back-up data restoration methods and device | |
| CN111459720B (en) | Mysql data recovery method based on transaction log | |
| CN108415990B (en) | Data quality monitoring method and device, computer equipment and storage medium | |
| CN115373889A (en) | Method and device for data comparison verification and data repair in data synchronization | |
| CN116560629A (en) | Serialization method, device, medium and equipment for data transmission object | |
| CN114385188A (en) | Code workload statistical method and device and electronic equipment | |
| CN114676054A (en) | Test data generation method, device, equipment, medium and product | |
| CN114356454A (en) | Account checking data processing method, account checking data processing device, account checking data storage medium and program product | |
| CN113076365A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| EP2980702A1 (en) | Method for enhancing the generation of a backup copy of data items of a distributed data structure, computer network for enhancing the generation of a backup copy of data items of a distributed data structure, program and computer program product | |
| CN111752927A (en) | Clone-based data form generation method, device, terminal equipment and medium | |
| CN112130849A (en) | Automatic code generation method and device | |
| US8326808B2 (en) | Information processing apparatus, and information processing method, program, and recording medium | |
| CN114461605B (en) | Transaction data multi-version implementation method, device and equipment of memory multi-dimensional database | |
| CN111240891A (en) | Data recovery method and device based on data consistency among multiple tables of database | |
| CN112685325B (en) | ETL software research and development test management method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |