CN111382198A

CN111382198A - Data recovery method, device, equipment and storage medium

Info

Publication number: CN111382198A
Application number: CN201811629215.4A
Authority: CN
Inventors: 张征; 陈战伟; 王鑫; 程哲慧; 李栋; 任志荣
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Shanxi Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Shanxi Co Ltd
Priority date: 2018-12-28
Filing date: 2018-12-28
Publication date: 2020-07-07
Anticipated expiration: 2038-12-28
Also published as: CN111382198B

Abstract

The invention discloses a data recovery method, a device, equipment and a storage medium, wherein the method comprises the following steps: determining at least one table associated with a predetermined field according to the predetermined field in the predetermined table; forming an association relation template between tables according to the association relation between the preset table and the at least one table; according to the incidence relation template, collecting and backing up table data associated with each value in at least one value of the preset field; and obtaining the table data associated with the target value in the at least one value from the backed-up table data for data restoration. According to the embodiment of the invention, the batch synchronization efficiency of the relational database can be improved, the cost is reduced, and the automatic regression test can be completed regularly and quantitatively without depending on manual intervention.

Description

Data recovery method, device, equipment and storage medium

Technical Field

The invention belongs to the field of computers, and particularly relates to a data recovery method, a data recovery device, data recovery equipment and a storage medium.

Background

It is well known that databases are important components that cannot be deleted. During software development, debugging and testing, programmers need to continuously test the current application to ensure the correctness, accuracy and robustness of the software. Most tests verify whether an application can perform the desired add-drop-and-modify operations on particular data within the database.

Some of the test data contents belong to non-renewable resources, for example, after a test for adding operation to one piece of data in a database is performed, target expected data already exists, which is often limited by a primary key, a unique index or a business rule, and the current data contents cannot be used for adding operation. Unless the current data content is deleted in a manual mode, or a piece of data content meeting the test requirement is searched for, or the data in the database is recovered. There are currently two ways to implement data recovery:

relational database batch recovery

The main stream relational database completes the batch application data restoration mainly by means of backup data batch synchronization or data snapshot, and the data sources are backup databases and logs. The method mainly aims to deal with database information safety and reserve client data to the maximum extent in sudden failures such as power failure and the like.

For example, a mainstream Oracle and MySQL database supports large-batch data synchronization through self application functions and internal or third-party tools such as SQLLOAD and goodenGate; oracle itself can respond to data that occurs within minutes or even hours in a contingency, depending on the underlying capabilities.

Batch recovery of non-relational database

The mainstream non-relational database comprises databases such as Hadoop, Hbase and MonggoDB, data is stored in a 1-to-N mode mainly through internal main and standby and emergency mechanisms, but when one data storage node fails to work and data is unavailable or lost, a management node can automatically start and execute a safety strategy, synchronize data of other data nodes and continuously keep a data persistence state of the data 1-to-N.

The data source of the non-relational database is from storage, a high availability mechanism is adopted, and the purpose of the method is mainly to deal with database information safety and reserve customer data to the greatest extent in sudden failures such as fault nodes, downtime and the like.

Thirdly, manually maintaining small amount of data synchronization

Whether the relational database or the non-relational database is adopted, a small amount of target data can be subjected to increasing, deleting, modifying and checking operations in a mode of manually compiling a data operation script, such as a Structured Query Language (SQL) script. And a small amount of target data is basically restored by reversely compiling a rollback script, backing up a local online script and the like.

The manual script data maintenance has strong operability, wide application range and strong flexibility, and is generally used for daily data maintenance. The method also comprises an application data reduction function in the application development and test processes.

The prior art has the following disadvantages:

1. the batch data recovery cost is high, and the demand of rapid test at any time on demand cannot be met: the problem that the content of the test data is unavailable due to the fact that the target data are changed can be solved by regularly performing data recovery on the test database and the research and development database, but in the face of large-scale industrial application, one test scene may involve parallel recovery of a plurality of databases, and the cost is high; meanwhile, the recovered data content is still unavailable after test use, and needs to search for another target or wait for next batch recovery.

2. The manual maintenance efficiency is low, and resources are difficult to share: the recovery of test data by means of manual scripts is a common practice in software development. And a programmer or a tester writes a backspacing script temporarily according to application and business scene cognition, so that the next debugging and use are facilitated. The temporarily written script is usually only aimed at one or a plurality of specific scenes, and once the test is completed, the script is usually invalidated and is rewritten next time.

3. Complexity issues of industry application data: either manual scripting or batch database data recovery face the complexity of industry application data. Such as: the time problem, yesterday, the test data that can be used in the last month cannot be used today or this month due to the limitation of the business rule, specifically as follows: when the discount is discounted by handling the discount in the current month, the data in the previous month cannot be used whether the data is recovered manually or by a database. At this point, the order date needs to be manually modified (UPDATE) to the date of the day of the month, which is a common "data making" phenomenon in the software industry.

Disclosure of Invention

Embodiments of the present invention provide a data reduction method, apparatus, device, and storage medium, which can solve the problems of low efficiency and high cost of batch synchronization of relational databases, and can timely and quantitatively complete auto-regression testing without manual intervention.

In a first aspect, an embodiment of the present invention provides a data restoring method, where the method includes:

determining at least one table associated with a predetermined field according to the predetermined field in the predetermined table;

forming an association relation template between tables according to the association relation between the preset table and the at least one table;

according to the incidence relation template, collecting and backing up table data associated with each value in at least one value of the preset field;

and obtaining the table data associated with the target value in the at least one value from the backed-up table data for data restoration.

In a second aspect, an embodiment of the present invention provides a data restoring apparatus, where the apparatus includes:

the association table determining module is used for determining at least one table associated with a preset field according to the preset field in the preset table;

the template forming module is used for forming an association relation template between the tables according to the association relation between the preset table and the at least one table;

the data acquisition module is used for acquiring and backing up table data associated with each value in at least one value of the preset field according to the association relation template;

and the data restoration module is used for acquiring the table data associated with the target value in the at least one value from the backed-up table data so as to restore the data.

In a third aspect, an embodiment of the present invention provides a data restoring apparatus, where the apparatus includes: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements the data reduction method.

In a fourth aspect, an embodiment of the present invention provides a computer storage medium, where computer program instructions are stored, and when the computer program instructions are executed by a processor, the data recovery method is implemented.

The data reduction method, the data reduction device, the data reduction equipment and the storage medium can perform data reduction through the table data associated with one or more target values from a specific scene, have high data reduction efficiency, can complete the data reduction of a test case in a short time, can be repeatedly used after testing, and solve the problems of low efficiency and high cost of batch synchronization of the relational database. Moreover, the incidence relation template is introduced to solve the complexity problem of industry data reduction, fields of all tables can be customized according to the requirements of specific scenes, automatic data reduction is realized, and reusability is greatly improved. Furthermore, large-scale mass background regression testing can be supported, and automatic regression testing can be completed regularly and quantitatively without depending on manual intervention after a testing scene and data reduction are established.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart illustrating a data reduction method according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating fields in a data table according to one embodiment of the invention;

FIG. 3 is a diagram illustrating an association relationship between fields, according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating an association between tables, according to an embodiment of the present invention;

FIG. 5 is a diagram illustrating fields in a data table according to another embodiment of the present invention;

FIG. 6 is a diagram illustrating an association relationship between fields according to another embodiment of the present invention;

FIG. 7 is a diagram illustrating an association relationship between tables according to another embodiment of the present invention;

FIG. 8 is a simplified diagram illustrating an inter-table association according to an embodiment of the present invention;

FIG. 9 is a flow chart illustrating a data reduction method according to another embodiment of the present invention;

FIG. 10 illustrates a flowchart of configuring an association relationship template, according to an embodiment of the invention;

FIG. 11 illustrates a flow diagram of gather list data according to an embodiment of the invention;

FIG. 12 shows a flow diagram of a template reduction process of one embodiment of the invention;

FIG. 13 is a schematic diagram of a data reduction apparatus according to an embodiment of the present invention;

fig. 14 is a schematic diagram illustrating a hardware structure of the data restoring apparatus according to the embodiment of the present invention.

Detailed Description

Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the present invention.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

In order to solve the problem of the prior art, embodiments of the present invention provide a data restoring method, apparatus, device, and computer storage medium. The following first describes a data reduction method provided in an embodiment of the present invention.

Fig. 1 is a schematic flow chart illustrating a data restoring method according to an embodiment of the present invention. As shown in fig. 1, the method includes:

s101, determining at least one table associated with the preset field according to the preset field in the preset table.

S102, forming an association relation template between the tables according to the association relation between the preset table and at least one table.

S103, collecting and backing up table data associated with each value in at least one value of the preset field according to the association relation template.

And S104, acquiring the table data associated with the target value in the at least one value from the backed-up table data to restore the data.

For example, after S103, a test is performed, and due to the test or other reasons, data changes occur, S104 may be performed to perform data restoration.

It should be noted that, in the embodiment of the present invention, automatic reduction of local data is implemented based on field association relations among data tables in a relational database. When a field value of a data table is known, all the related object contents related to the previous transaction in the data table affected by the transaction in the whole database can be automatically restored according to the field value. Such as: in the mobile communication industry, after a product change test is carried out on a mobile phone number, the content of a data line in a series of lists related to the mobile phone number in the current database environment is restored, and further secondary test or other test work can be conveniently carried out.

In the embodiment of the invention, firstly, starting from a specific scene, data reduction is carried out through table data associated with one or more target values, the data reduction efficiency is high, the data reduction of a test case can be completed within a short time (about 2 minutes), the test case can be repeatedly used after testing, and the problems of low efficiency and high cost of batch synchronization of the relational database are solved. And secondly, an incidence relation template is introduced to solve the complexity problem of industry data reduction, fields of all tables can be customized according to the requirements of specific scenes, automatic data reduction is realized, reusability is greatly improved, and the complexity problem of industry application data is solved. And finally, the data can be reused and customized by special field characteristics, large-scale mass background regression testing can be supported, and automatic regression testing can be completed regularly and quantitatively without manual intervention after a testing scene and data reduction are established.

In one embodiment of the present invention, S101 includes:

taking a predetermined field as a starting field; searching a field associated with the initial field, and taking a table where the field is located as a table to be processed; taking each field in the table to be processed as a new initial field, and returning to the step of executing the search until a first preset condition is met; and taking all the obtained tables to be processed as at least one table.

Wherein the first predetermined condition comprises: and finding out the field associated with the initial field or obtaining the table to be processed to meet the requirement of the test scene.

It should be noted that, in the actual production application work of the database, based on factors such as efficiency and business complexity, the design and development do not completely use the main foreign key of the database to maintain the data consistency, but maintain the business data consistency in the business logic code through the upper WEB or middleware application. FIG. 2 shows a diagram of fields in a data table, according to an embodiment of the invention. As shown in fig. 2, the relationships between the fields are distinguished by the shapes of the outer frames of the respective fields, but of course, the relationships between the fields may also be distinguished by the colors of the respective fields in the table. In fig. 2, field 1 in data table a and field 2 in data table B are both square outer frames, i.e. field 1 in data table a and field 2 in data table B have an association relationship. Similarly, field 1 in data table B and field 1 in data table C have an association, and field 3 in data table C and field 1 in data table D have an association.

According to the field association relationship in each table in fig. 2, a schematic diagram of the association relationship between the fields as shown in fig. 3 can be obtained. As shown in fig. 3, a1 has an association with B2, B1 has an association with C1, C3 has an association with D1, a1 represents field 1 in data table a, B2 represents field 2 in data table B, and the meaning of B1, C1, C3, D1 can be analogized.

Fig. 2 enumerates the association relationship of 4 data tables, that is, after obtaining the instance value of a1 (e.g. the user PHONE number PHONE _ NO), one or a group of B2 instance values can be obtained immediately through SQL statements, and one or a group of one-to-one, one-to-many or many-to-many relationships are determined by a1 and B2.

After obtaining B2 based on A1, B1 can be directly obtained in a data B table of B2, and then one or more groups of C1 are obtained. Similarly, a list of all D1 fields may be derived by relying on a list of all derived C1 fields.

FIG. 4 is a diagram illustrating the table associations according to an embodiment of the present invention, wherein the A table data row is acquired in dependence on A1, and the B table data row is acquired in dependence on A1, both of which are acquired directly; the data acquisition of B2 obtained by the data row of the C table A1 and the data acquisition of C3 in the data content of C1 obtained by the data row of the D table B1 are both indirect acquisition. In fig. 4, the solid line represents the direct acquisition relationship, and the broken line represents the indirect acquisition relationship.

In practical application, the data model design is designed according to specific application requirements and mostly evolves in increments, so that the association relation result of the transaction type data table in the general industry can have a graph-like structure. The embodiment of the invention converts the graphic association relationship into the binary tree-shaped association relationship.

FIG. 5 shows a schematic diagram of fields in a data table according to another embodiment of the present invention. As shown in fig. 5, based on the four tables of the user basic information (data table a), the user subscription information (data table B), the user subscription attribute information (data table C), and the user attribute configuration information table (data table D), the service operation rule record table X is added according to the service requirement, and the association relationship between the fields is shown in fig. 6. In fig. 6, the association relationship of fields is complex and redundant, and there are several ways to obtain B, C, D data row contents of the table in the case of knowing a field instance value of data a1, as shown in fig. 7, if it is known that a1 needs to obtain B table data, it can be obtained by: more than three ways of A1-B2, A1-X1-B1, A1-X1-C3-B1 and the like are associated and obtained, if the B table is obtained through the three ways, the data redundancy of the collected table and repeated operation during data reduction are caused, the research and development site cost is increased, and the processing efficiency is reduced.

Based on the above problem, in one embodiment of the present invention, the method further comprises:

after obtaining a plurality of tables to be processed, judging whether repeated tables exist in the plurality of tables to be processed; if there are duplicate tables, one of the duplicate tables is retained and the other tables in the duplicate table are deleted.

The incidence relation template is a binary tree-shaped incidence relation template.

In the embodiment of the invention, only one table is reserved for the repeated table, and other tables are deleted, for example, if the table A and the table B are repeated, one table in the table A and the table B is reserved, and the other table is deleted. The problems of multiple association modes, redundant storage and repeated restoration due to the occurrence of repeated tables are avoided, so that the association complexity is reduced, and the recovery efficiency is improved. As shown in fig. 8, duplicate tables are eliminated, and the association relationship between tables is greatly simplified.

In one embodiment of the present invention, S103 includes:

for each of the at least one value, performing a predetermined step to obtain table data associated with each of the at least one value.

Wherein the predetermined step comprises:

taking a current value to be processed in at least one value as a starting value; sequentially collecting at least one table to be collected associated with the initial value according to the sequence of the tables in the association relation template; taking the field value in each table to be acquired as a new initial value, and returning to execute at least one table to be acquired associated with the new initial value until a second preset condition is met; and taking the obtained data in all the tables to be acquired as the table data associated with the current value to be processed.

Wherein the second predetermined condition comprises: there is no table to be acquired associated with the start value within the predetermined acquisition range.

As an example, all the related data line contents of A, B, C and D tables can be obtained based on the field value of A1, and the data line contents can be used for subsequent data recovery operation after being recorded and backed up.

For example: when table data associated with a specific mobile phone number is acquired, the mobile phone number is used as an initial value, and a user ordering information table associated with the mobile phone number is acquired according to the sequence of the table in the association relation template; secondly, taking the field value in the user ordering information table as a new initial value, and collecting a user ordering attribute information table associated with the new initial value; and thirdly, taking the field value in the user subscription attribute information table as a new initial value, and collecting the user attribute configuration information table associated with the new initial value. Therefore, according to the sequence of the table in the association relation template, the data in the user ordering information table, the user ordering attribute information table and the user attribute configuration information table are collected in sequence.

In one embodiment of the present invention, S104 includes:

when data reduction is performed, table data within a predetermined reduction range is replaced with table data associated with a target value.

In one embodiment of the present invention, S104 includes:

the data of the table within the predetermined restoration range is deleted by the deletion instruction, and the table data associated with the target value is inserted by the insertion instruction.

Fig. 9 is a schematic flow chart illustrating a data restoring method according to another embodiment of the present invention. As shown in fig. 9, the method includes:

s201, configuring an association relation template based on the association relation of the fields.

Wherein, S201 includes:

and S2011, selecting and recording a predetermined table and a predetermined field according to the service test and automatic test requirements. Such as: in the mobile industry, a data table and a data line are directly and indirectly associated from a mobile phone number under a user basic table, and the data can meet the service test requirement of more than about 95 percent after being recorded and restored.

S2012, the mapping relation is combed, the predetermined field is used as a starting field, and all the tables directly related to the starting field are searched in the database and recorded (or backed up). And eliminating the recorded tables during recording. The priority order of the next data acquisition is defined during recording.

S2013, taking the field in the data table obtained in step S2012 as a new start field, and repeating the actions of the lookup table in step S2012 until all the fields in the data table obtained in step S2012 are looked up. And eliminating the recorded tables during recording.

And S2014, taking the field in the data table obtained in S2013 as a new initial field, and repeating the action of the lookup table in S2012 until the data result obtained in the step S2013 meets the requirement of the test scene. And eliminating the recorded data tables during recording.

S2015, persisting the results of S2014, for predetermined fields of a predetermined table, without then having to perform the S201 operation again.

And then, completing configuration of the association relation template based on the preset field, and delivering a relation data example result between the service test dimension data table with a binary tree result and the field, namely the association relation template. The incidence relation template does not depend on specific values of a preset table and a preset field, and any field value which is not empty can be used for directly or indirectly positioning data rows in a full data table required by a data reduction scene by using the incidence relation template.

FIG. 10 is a flow diagram illustrating configuring an association relationship template, according to an embodiment of the invention. As shown in fig. 10, the template configuration application is responsible for storing the business association relationship in the artificially-sorted target database and entering the configuration code table of the management database, and the configuration at this time is mainly based on pseudo SQL, that is, some variables, such as operation time, operation flow and the like, which need to be dynamically acquired only when the acquisition execution or the reduction execution exists in the assignment part of the SQL statement.

S202, collecting data in the table based on at least one field value and the association relation template, and backing up the collected data.

The delivery of S202 is specific data of a data row in the full data table required for the data recovery scenario, and is used to collect data records before operation in the data recovery scenario. Different start tables and field values in the start tables, different data are collected.

Wherein, S202 includes:

s2021: an inventory target template and at least one valid field value of the predetermined field are input to the system application. The target template is used for defining the range of data acquisition and data reduction, and the effective field value is the data starting point of a specific reduction operation.

S2022: the system automatically takes the effective field value as a starting value according to the effective field value, and collects all the contents of the data row of the table associated with the starting value. And sequentially collecting the data of each table associated with the initial value according to the sequence of the tables in the association relation template, and backing up.

S2023: the operation in step S2021 is repeated with the field value in the table obtained in step S2022 as the new initial value until the data content obtained in step S2022 is complete and complete.

S2024: and (4) repeating the step (S2022) by taking the data result obtained in the step (S2023) as a new initial value until all the configuration relations in the association relation template in the step (S2023) are processed.

S2025: the result of S2024 is persistently stored, and the S202 operation need not be performed every time the test scenario data is restored later.

And then, acquiring data in the table based on at least one field value and the association relation template, completing backup of the acquired data, and delivering a result set recorded before the service test operation of the binary tree result. The result set does not depend on any test scene, and tests in all operation table ranges can be repeatedly used. Such as: in the mobile communication industry, after the operation of S202 is completed by the mobile phone number under the user basic table, the service test requirement of more than 95 percent can be met.

FIG. 11 shows a flow diagram of the acquisition table data of one embodiment of the present invention. As shown in fig. 11, the template collection application is mainly composed of two application entities, namely, a generic variable pool and a pseudo SQL execution block, the generic variable pool is used as a generic access area to sequentially record encountered pseudo SQL variables K-V in a key-value pair manner, and the pseudo SQL execution block is used to dynamically obtain the current values of the variables in the pseudo SQL according to configured constants, environment variables or SQL statements of the data model. And circulating the steps until all the required pseudo SQL values are acquired in the acquisition period. After the collection of the pseudo SQL values is completed, the model collection application sequentially puts Deleted (DELETE) pseudo SQL and Inserted (INSERT) pseudo SQL into a table in a mode of processing data instances by a template according to the management data model configuration.

And S203, executing the operation of restoring the database data based on the target field value, the incidence relation template and the backed-up data.

S203 delivers a pre-test data preparation phase for all data content of the test data restore. This step is generated in a relational database, and test work can be performed based on the data backed up at S202. The steps can be repeatedly executed for any time, such as: after executing S203, one mobile phone number may be used for multiple and repeated tests to verify the correctness of the system application in the pre-online test environment or the correctness of the system in the severe post-online production environment.

Wherein, S203 includes:

s2031, inputting the inventory target template and the valid target value to the system application. The template is used to define the data collection and reduction range, the target value is the data starting point of a specific reduction operation, the target value is the value in at least one valid field value in S2021, and the number of the target values may be one or more.

S2032, the system automatically extracts the table data associated with the target value from the data backed up in S202, replaces the KEY variables in the pseudo SQL deleted by data one by one, generates the deleted SQL, and executes the deleted SQL.

The purpose of deleting SQL is to delete historical data that was last tested or otherwise generated.

S2033, extracting the table data associated with the target value from the data backed up in S202, replacing the KEY variables in the pseudo SQL restored with the data one by one, generating a restored SQL, and executing the restored SQL.

And S2034, after all the steps S2033 are executed and all the steps are correct, submitting the database and feeding back success information to the upstream of the system. And S2033, after all execution is finished and errors occur, the database rolls back, and failure information is fed back to the upstream of the system for positioning BUGs (BUG).

And then, based on the target value, the association relation template and the backed-up data, completing the restoration of the operated database data and delivering the relational database content required by a test for restoration. S203 repeats the calling, executing, and delivering every time a specific test is performed.

FIG. 12 is a flow diagram illustrating a template reduction process, according to an embodiment of the invention. As shown in fig. 12, the template reduction processing application is mainly composed of two application entities, namely, a generic variable pool and a pseudo SQL execution block, the generic variable pool is used as a generic access area to sequentially record encountered pseudo SQL variables K-V in a key-value pair manner, and the pseudo SQL execution block is used to dynamically obtain current values of variables in the pseudo SQL according to configured constants, environment variables or SQL statements of the data model. And circulating the steps until all the needed pseudo SQL values are acquired during the restoration processing.

According to the embodiment of the invention, firstly, from the service logic Relationship, after template design and record storage, data required by the test can be selectively and automatically restored.

Secondly, based on the business logic relation of data tables and fields in the relational database, the application design, the flow design and the model design of a data reduction scene are realized, the problem of low efficiency of data reduction operation is solved, the problem of complexity of data reduction of different data tables in different test scenes is solved, the test efficiency is improved, and the method can be used for automatic regression test of an automatic and large-scale system.

The embodiment of the invention at least comprises the following four advantages:

1. the method and the device solve the problems of low efficiency and high cost of batch synchronization of the relational database, and the embodiment of the invention starts from the local data of a single database in a specific test scene, has high data reduction efficiency, can complete the full data reduction of one test case within 2 minutes, and can be repeatedly used after the test.

2. The embodiment of the invention adopts an independent data maintenance system, automatically generates and delivers DELETE scripts and INSERT scripts for data reduction in the test case, can use the same test number as required, and has no exception as long as the test time (within 2 minutes) does not occur simultaneously.

3. The method and the device solve the complexity problem of the industrial application data, solve the complexity problem of the industrial application data in a multi-template external variable and pseudo SQL (structured query language) mode, can customize special fields of each data table according to specific scene requirements, and have automatic data generation and greatly improved reusability.

4. Based on the characteristic that the data of the proposal can be reused and the special field is customized, the embodiment of the invention can support large-scale mass background regression testing, and the automatic regression testing can be completed regularly and quantitatively without depending on manual intervention after the test scene and the data reduction are established, which cannot be provided by the traditional scheme.

Fig. 13 is a schematic structural diagram of a data restoring apparatus according to an embodiment of the present invention. As shown in fig. 13, the apparatus 300 includes:

an association table determining module 301, configured to determine, according to a predetermined field in a predetermined table, at least one table associated with the predetermined field.

The template forming module 302 is configured to form an association relationship template between tables according to an association relationship between a predetermined table and at least one table.

A data collecting and backing up module 303, configured to collect and back up table data associated with each of at least one value of the predetermined field according to the association relation template.

And a data restoring module 304, configured to obtain, from the backed-up table data, the table data associated with the target value of the at least one value, so as to perform data restoration.

In one embodiment of the present invention, the association table determining module 301 includes:

a first starting field determining module, configured to use a predetermined field as a starting field;

and the table lookup module is used for looking up the field associated with the initial field and taking the table where the field is positioned as the table to be processed.

And the second starting field determining module is used for taking each field in the table to be processed as a new starting field.

And the table lookup module is further used for repeatedly executing the lookup step until the first preset condition is met.

And the first table determining module is used for taking all the obtained tables to be processed as at least one table.

In one embodiment of the invention, the first predetermined condition comprises: and finding out the field associated with the initial field or obtaining the table to be processed to meet the requirement of the test scene.

In one embodiment of the present invention, the apparatus 300 further comprises:

the judging module is used for judging whether repeated tables exist in the multiple to-be-processed tables after the multiple to-be-processed tables are obtained;

and the duplication removing module is used for keeping one table in the duplicated tables and deleting other tables in the duplicated tables when the duplicated tables exist.

In an embodiment of the present invention, the association relationship template is a binary tree-like association relationship template.

In one embodiment of the present invention, the data collection and backup module 303 comprises:

a data acquisition module for performing a predetermined step for each of the at least one value to obtain table data associated with each of the at least one value.

Wherein, the data acquisition module includes:

and the first starting value determining module is used for taking the current value to be processed in the at least one value as the starting value.

And the table acquisition module is used for sequentially acquiring at least one table to be acquired which is associated with the initial value according to the sequence of the tables in the association relation template.

And the second starting value determining module is used for taking the field value in each table to be acquired as a new starting value.

And the table acquisition module is also used for repeatedly executing at least one table to be acquired which is associated with the new starting value until a second preset condition is met.

And the second table determining module is used for taking the obtained data in all the tables to be acquired as the table data associated with the current value to be processed.

In one embodiment of the invention, the second predetermined condition comprises: there is no table to be acquired associated with the start value within the predetermined acquisition range.

In one embodiment of the present invention, the data restoring module 304 includes:

and the data replacement module is used for replacing the table data in the preset reduction range with the table data associated with the target value.

In one embodiment of the invention, the data replacement module comprises:

and the data deleting module is used for deleting the data of the table in the preset restoring range through the deleting instruction.

And the data insertion module is used for inserting the table data associated with the target value through the insertion instruction.

In one embodiment of the invention, a data collection and backup module collects table data associated with each of at least one value of a predetermined field in a relational database.

And the data reduction module is used for reducing the data in the relational database according to the table data associated with the target value.

The data reduction device may include a processor 401 and a memory 402 storing computer program instructions.

Specifically, the processor 401 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured as one or more Integrated circuits implementing embodiments of the present invention.

Memory 402 may include mass storage for data or instructions. By way of example, and not limitation, memory 402 may include a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, tape, or Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 402 may include removable or non-removable (or fixed) media, where appropriate. The memory 402 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 402 is a non-volatile solid-state memory. In a particular embodiment, the memory 402 includes Read Only Memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory or a combination of two or more of these.

The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement any of the data recovery methods in the above embodiments.

In one example, the data reduction device may also include a communication interface 403 and a bus 410. As shown in fig. 14, the processor 401, the memory 402, and the communication interface 403 are connected by a bus 410 to complete communication therebetween.

The communication interface 403 is mainly used for implementing communication between modules, apparatuses, units and/or devices in the embodiments of the present invention.

Bus 410 includes hardware, software, or both to couple the components of the data reduction device to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 410 may include one or more buses, where appropriate. Although specific buses have been described and shown in the embodiments of the invention, any suitable buses or interconnects are contemplated by the invention.

The data restoring device may execute the data restoring method in the embodiment of the present invention, so as to implement the data restoring method and apparatus described in conjunction with fig. 1 and 13.

In addition, in combination with the data restoring method in the foregoing embodiments, the embodiments of the present invention may provide a computer storage medium to implement. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement any of the data reduction methods in the above embodiments.

It is to be understood that the invention is not limited to the specific arrangements and instrumentality described above and shown in the drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present invention.

The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

It should also be noted that the exemplary embodiments mentioned in this patent describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.

As described above, only the specific embodiments of the present invention are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.

Claims

1. A method of data reduction, comprising:

2. The method of claim 1, wherein determining at least one table associated with a predetermined field of a predetermined table according to the predetermined field comprises:

taking the predetermined field as a starting field;

searching a field associated with the initial field, and taking a table where the field is located as a table to be processed;

taking each field in the table to be processed as a new initial field, and returning to the step of executing the search until a first preset condition is met;

and taking all the obtained tables to be processed as the at least one table.

3. The method of claim 2, wherein the first predetermined condition comprises: and finding out fields associated with the initial fields or obtaining the table to be processed to meet the requirements of the test scene.

4. The method of claim 2, further comprising:

after obtaining a plurality of tables to be processed, judging whether repeated tables exist in the plurality of tables to be processed;

and if the repeated tables exist, one table in the repeated tables is reserved, and other tables in the repeated tables are deleted.

5. The method according to claim 1, wherein the association relation template is a binary tree-like association relation template.

6. The method of claim 1, wherein collecting table data associated with each of at least one value of the predetermined field according to the association relationship template comprises:

performing a predetermined step for each of the at least one value to obtain table data associated with each of the at least one value;

wherein the predetermined step comprises:

taking a current value to be processed in the at least one value as a starting value;

sequentially collecting at least one table to be collected associated with a starting value according to the sequence of the tables in the association relation template;

taking the field value in each table to be acquired as a new starting value, and returning to execute at least one table to be acquired associated with the new starting value until a second preset condition is met;

and taking the obtained data in all the tables to be acquired as the table data associated with the current value to be processed.

7. The method of claim 6, wherein the second predetermined condition comprises: there is no table to be acquired associated with the start value within the predetermined acquisition range.

8. The method of claim 1, wherein,

and replacing the table data in the preset reduction range with the table data associated with the target value when the data reduction is carried out.

9. The method of claim 8, wherein,

deleting data of the table within a predetermined restoration range by a deletion instruction, and inserting the table data associated with the target value by an insertion instruction.

10. The method of claim 1, wherein,

collecting table data associated with each of at least one value of the predetermined field in a relational database;

and the number of the first and second electrodes,

and restoring the data in the relational database according to the table data associated with the target value.

11. A data recovery apparatus, comprising:

12. A data reduction apparatus, characterized in that the apparatus comprises: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements a data reduction method as claimed in any one of claims 1-10.

13. A computer storage medium having computer program instructions stored thereon, which when executed by a processor implement the data reduction method of any one of claims 1 to 10.