CN114416845A - Big data testing method and device, electronic equipment and storage medium - Google Patents
Big data testing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114416845A CN114416845A CN202210061230.3A CN202210061230A CN114416845A CN 114416845 A CN114416845 A CN 114416845A CN 202210061230 A CN202210061230 A CN 202210061230A CN 114416845 A CN114416845 A CN 114416845A
- Authority
- CN
- China
- Prior art keywords
- data
- service data
- test
- target
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the technical field of automatic testing, and provides a big data testing method, a device, electronic equipment and a storage medium. Through data verification of a simulation test service system data flow- > ODS layer- > DW layer- > ADS layer- > multi-bin upper interface layer- > application system, a complete data flow closed loop is formed, and test efficiency and test quality of big data are guaranteed.
Description
Technical Field
The invention relates to the technical field of automatic testing, in particular to a big data testing method and device, electronic equipment and a storage medium.
Background
Big data refers to a collection of data that cannot be captured, managed and processed with conventional software tools in an affordable time frame. The prior art tests for large data platforms include manual tests and automated tests.
In the process of implementing the invention, the inventor finds that when a large data platform is tested in a manual mode, the service system is continuously updated in an iterative manner, and the service data changes every day, so that the testing efficiency of the manual testing is low, and in the process of testing mass data, data are easily missed manually, and the full-flow testing is not verified for enough time, so that the testing quality is difficult to ensure. When the large data platform is tested in an automatic mode, because the automatic test aiming at the large data platform is usually carried out, an automatic test framework of the large data platform is specially designed aiming at a certain specific system, the transportability is poor, the large data platform can not be used for other large data platforms, and the universality is not high.
Disclosure of Invention
In view of the above, it is necessary to provide a big data testing method, device, electronic device and storage medium, which can improve testing efficiency and ensure testing quality.
A first aspect of the present invention provides a big data testing method, including:
when detecting that target service data are stored in a service system database, extracting the target service data to a plurality of ODS layers;
when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the multi-bin ODS layer, extracting the service data of the multi-bin ODS layer to a multi-bin DW layer;
performing data test on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case;
responding to the instruction that the data test passes, and pushing the service data extracted from the ADS layer to the upper interface layer of the plurality of bins to a target application system;
and carrying out consistency check test on the service data pushed to the target application system and the target service data to obtain a consistency check test result.
According to an optional embodiment of the present invention, the extracting the target service data to the number bin ODS layer comprises:
acquiring a plurality of data fields of the target service data;
generating an ODS test script according to the plurality of data fields;
calling the ODS test script to obtain a binlog log of the target service data;
controlling the analysis progress of the binlog according to the identifier of the binlog log;
sending the analyzed data to a Kafka queue;
and extracting the data in the Kafka queue to the number bin ODS layer based on the corresponding relation configured in the number bin ODS layer in advance.
According to an alternative embodiment of the invention, the method further comprises:
when the data table structure of the target service data is inconsistent with the data table structure of the service data extracted to the ODS layer of the bins, deleting the service data of the ODS layer of the bins;
deriving a table building statement of the target service data from a database of the service system;
executing the table building statement in the number-bin ODS layer such that the target service data is synchronized to the number-bin ODS layer.
According to an optional embodiment of the present invention, the generating process of the new successful test case includes:
acquiring a successful test case set in an original test case set;
searching K nearest neighbor successful test cases of each successful test case in the successful test case set;
selecting a target successful test case from the successful test case set each time;
generating a random integer between 1 and K, and generating candidate test cases by using a test case calculation model according to the target successful test case and the random integer;
testing the candidate test case;
and responding to the instruction that the candidate test case passes the test, and determining the candidate test case as the new successful test case.
According to an optional embodiment of the present invention, the generating a candidate test case by using a test case calculation model according to the target successful test case and the random integer includes:
generating a case array based on K nearest neighbor successful test cases of the target successful test case;
acquiring an array index corresponding to the random integer in the case array and a target nearest neighbor successful test case corresponding to the array index;
calculating a difference factor between the target successful test case and the target nearest neighbor successful test case;
generating a test seed by using a test case calculation model;
and generating candidate test cases based on the target successful test case, the test seeds and the difference factor.
According to an optional embodiment of the present invention, the test case calculation model is: ti + rand (0, 1) × tr-ti |, wherein r represents a random integer, tr represents a target nearest neighbor successful test case corresponding to an array index which is the same as the random integer, and ti represents a selected ith target successful test case.
According to an optional embodiment of the present invention, the performing a consistency check test on the service data pushed to the target application system and the target service data to obtain a consistency check test result includes:
acquiring a page ID for producing the target service data in the service system;
using SQL query statements to query the business data of the target application system to obtain an SQL query result;
acquiring the numerical value of each row in the SQL query result and a result ID corresponding to the numerical value of each row;
acquiring a page ID which is the same as the result ID of each row;
comparing whether the numerical value corresponding to the result ID of each row is consistent with the service data of the same row corresponding to the same page ID;
and when the numerical values corresponding to the result IDs of all the rows are consistent with the service data of all the rows corresponding to the same page ID, the consistency verification test result is that the consistency verification test is passed.
A second aspect of the present invention provides a big data test apparatus, the apparatus comprising:
the system comprises a first extraction module, a data processing module and a data processing module, wherein the first extraction module is used for extracting target service data to a multi-bin ODS layer when detecting that the target service data are stored in a service system database;
a second extraction module, configured to extract the service data in the ODS layer of the several bins to the DW layer when a data table structure of the target service data is consistent with a data table structure of the service data extracted to the ODS layer of the several bins;
the data testing module is used for carrying out data testing on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case;
the data pushing module is used for responding to the instruction that the data test passes and pushing the service data extracted from the ADS layer of the plurality of bins to the interface layer of the upper layer of the plurality of bins to a target application system;
and the verification test module is used for performing consistency verification test on the service data pushed to the target application system and the target service data to obtain a consistency verification test result.
A third aspect of the invention provides an electronic device comprising a processor for implementing the big data test method when executing a computer program stored in a memory.
A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the big data test method.
In summary, the big data testing method, the device, the electronic device and the storage medium according to the present invention extract the target service data in the service system to the multi-bin ODS layer, extract the service data of the multi-bin ODS layer to the multi-bin DW layer when it is ensured that the data table structure of the target service data is consistent with the data table structure of the service data extracted to the multi-bin ODS layer, improve the testing quality, perform data testing on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case, push the service data extracted from the multi-bin ADS layer to the multi-bin upper interface layer to the target application system in response to the instruction that the data testing passes, ensure the consistency between the incremental data of the service system and the incremental data of the multi-bin upper interface, perform consistency check testing on the service data pushed to the target application system and the target service data, and obtaining a consistency check test result. The method has the advantages that the total test is executed through the service data of the service system, the change speed of the service system is met, the test efficiency is improved, a complete data transfer closed loop is formed through the data verification of the simulation test service system data flow- > ODS layer- > DW layer- > ADS layer- > digital cabin upper interface layer- > application system, and the test quality of big data is further guaranteed.
Drawings
Fig. 1 is a flowchart of a big data testing method according to an embodiment of the present invention.
Fig. 2 is a structural diagram of a big data testing apparatus according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a detailed description of the present invention will be given below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
The big data testing method provided by the embodiment of the invention is executed by the electronic equipment, and accordingly, the big data testing device runs in the electronic equipment.
Example one
Fig. 1 is a flowchart of a big data testing method according to an embodiment of the present invention. The big data testing method specifically comprises the following steps, and the sequence of the steps in the flow chart can be changed and some steps can be omitted according to different requirements.
And S11, when detecting that the target service data is stored in the service system database, extracting the target service data to a multi-bin ODS layer.
The business data produced by the business system is stored in a database of the business system, and the business data refers to the data which is produced by the business system in the actual production environment and is related to the business. For the convenience of the following description, the service data stored in the service system database is referred to as target service data.
The big data testing platform is in butt joint with one or more service systems so as to carry out full testing on the service data of the one or more service systems.
The big data test platform can actively detect whether the target service data is stored in the database of the service system in real time, and the service system can also actively report the target service data after storing the target service data in the database.
The big data testing platform can extract the target service data to the multi-bin ODS layer through the installed Jerner testing tool, and therefore testing is conducted.
The multiple bins of the big data testing platform can comprise 4 layers of architectures, namely an ODS layer, a DW layer, an ADS layer and an upper interface layer.
In an optional embodiment, the extracting the target service data into a number bin ODS layer includes:
acquiring a plurality of data fields of the target service data;
generating an ODS test script according to the plurality of data fields;
calling the ODS test script to obtain a binlog log of the target service data;
controlling the analysis progress of the binlog according to the identifier of the binlog log;
sending the analyzed data to a Kafka queue;
and extracting the data in the Kafka queue to the number bin ODS layer based on the corresponding relation configured in the number bin ODS layer in advance.
Due to the fact that business systems are various and databases used by the business systems are different, time and labor are consumed in the process of loading target business data to the multi-bin ODS layer. The ODS test script is generated according to a plurality of data fields of the target service data, the ODS test script can be used for automatically extracting data, and the ODS test script is suitable for various service systems and various databases of the service systems.
In addition, the analyzed identifier of the binlog log progress is stored in the Redis, and a tester can switch the analysis progress of the binlog by changing the corresponding numerical value in the Redis, so that the flexibility is improved. When the analysis progress of the binlog is too fast, the corresponding numerical value in the Redis can be reduced, so that the analysis progress of the binlog is reduced, when the analysis progress of the binlog is too slow, the corresponding numerical value in the Redis can be increased, the analysis progress of the binlog is improved, and therefore the target business data can be effectively prevented from being lost in the loading process, and the safety of loading the target business data is improved.
And S12, when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the ODS layer of the number of bins, extracting the service data of the ODS layer of the number of bins to the DW layer of the number of bins.
After extracting the target service data to the multi-bin ODS layer, the big data testing platform firstly judges whether the data table structure of the target service data is consistent with the data table structure of the service data extracted to the multi-bin ODS layer. And extracting the service data of the multi-bin ODS layer to the multi-bin DW layer only when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the multi-bin ODS layer.
In an optional embodiment, the method further comprises:
when the data table structure of the target service data is inconsistent with the data table structure of the service data extracted to the ODS layer of the bins, deleting the service data of the ODS layer of the bins;
deriving a table building statement of the target service data from a database of the service system;
executing the table building statement in the number-bin ODS layer such that the target service data is synchronized to the number-bin ODS layer.
The big data test platform analyzes the table building sentence to obtain the table name of the table to be built, the field name of the table header field, the field type and the field format, wherein the field type comprises a character type, a numerical value type and the like, and the field format is the format of the value corresponding to the field.
By exporting the table building statement of the target service data and executing the table building statement in the multi-bin ODS layer, the target service data can be quickly synchronized into the multi-bin ODS layer.
And S13, performing data test on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case.
And the big data test platform extracts the service data of the multi-bin ODS layer to the multi-bin DW layer through an ETL tool, and extracts the service data of the multi-bin DW layer to the multi-bin ADS layer through the ETL tool.
The test case set generally comprises successful test cases and failed test cases, and in order to improve the test quality of the data test, the more successful test cases are, the better is. The big data test platform can judge the number of successful test cases firstly, when the number is lower than a preset number threshold value, a new successful test case can be generated based on the successful test cases in the test case set firstly, the new generated test cases are combined into the test case set to obtain a new test case set, and therefore data test is conducted on business data on the basis of the new test case set on the multi-cabin ADS layer.
In an optional embodiment, the generating process of the new successful test case includes:
acquiring a successful test case set in an original test case set;
searching K nearest neighbor successful test cases of each successful test case in the successful test case set;
selecting a target successful test case from the successful test case set each time;
generating a random integer between 1 and K, and generating candidate test cases by using a test case calculation model according to the target successful test case and the random integer;
testing the candidate test case;
and responding to the instruction that the candidate test case passes the test, and determining the candidate test case as the new successful test case.
The Euclidean distance between every two successful test cases can be calculated, the Euclidean distances are sorted from small to large for any successful test case, and the successful test cases corresponding to the first K Euclidean distances in the sorting process are used as K neighbor successful test cases of any successful test case.
Random integers between 1 and K may be generated using a random number generation algorithm such that candidate test cases are generated using a test case computation model based on the generated random integers. In order to determine that the generated candidate test case is a successful test case, the candidate test case is tested first.
In an optional embodiment, the generating a candidate test case by using a test case calculation model according to the target successful test case and the random integer includes:
generating a case array based on K nearest neighbor successful test cases of the target successful test case;
acquiring an array index corresponding to the random integer in the case array and a target nearest neighbor successful test case corresponding to the array index;
calculating a difference factor between the target successful test case and the target nearest neighbor successful test case;
generating a test seed by using a test case calculation model;
and generating candidate test cases based on the target successful test case, the test seeds and the difference factor.
Based on the distance between the nearest neighbor successful test case and the target successful test case, the nearest neighbor successful test case with small distance is preferentially written into the case array according to the distance sequence, and the nearest neighbor successful test case with large distance is finally written into the case array. After the K nearest neighbor successful test cases are successfully written into the case array, the array indexes correspond to the nearest neighbor successful test cases one to one.
The test case computation model may be a rand function, generating a pseudo-random number between 0 and 1 as a test seed.
In an alternative embodiment, the test case calculation model may be represented by the following formula: ti + rand (0, 1) × tr-ti |, wherein r represents a random integer, tr represents a target nearest neighbor successful test case corresponding to an array index which is the same as the random integer, and ti represents a selected ith target successful test case.
And S14, responding to the instruction that the data test passes, and pushing the service data extracted from the ADS layer to the upper interface layer of the plurality of bins to a target application system.
After the data test of the service data of the ADS layer of the plurality of bins passes, the ETL tool is used for extracting the service data of the ADS layer of the plurality of bins to an interface layer at the upper layer of the plurality of bins. The upper interface layer of the several bins corresponds to one or more application systems, such as BI, HRP, etc.
The big data test platform traverses the service data extracted to the upper layer interface layer of the storage bins, and notifies the upper layer interface layer of the storage bins to push the service data to a target application system after verifying the correctness of the service data of the upper layer interface of the storage bins.
And S15, performing consistency check test on the service data pushed to the target application system and the target service data to obtain a consistency check test result.
The consistency check test result may include that the consistency check test passes and the consistency check test fails.
In an optional embodiment, the performing a consistency check test on the service data pushed to the target application system and the target service data to obtain a consistency check test result includes:
acquiring a page ID for producing the target service data in the service system;
using SQL query statements to query the business data of the target application system to obtain an SQL query result;
acquiring the numerical value of each row in the SQL query result and a result ID corresponding to the numerical value of each row;
acquiring a page ID which is the same as the result ID of each row;
comparing whether the numerical value corresponding to the result ID of each row is consistent with the service data of the same row corresponding to the same page ID;
and when the numerical values corresponding to the result IDs of all the rows are consistent with the service data of all the rows corresponding to the same page ID, the consistency verification test result is that the consistency verification test is passed.
And when the numerical value corresponding to the result ID of any row is inconsistent with the service data of all rows corresponding to the same page ID, the consistency verification test result is that the consistency verification test fails.
It should be emphasized that, in order to further ensure the privacy and security of the target service data in the service system database and the service data of each layer in the bins, the service data of each layer in the bins may also be stored in the nodes of a block chain.
The big data testing method provided by the invention can execute full test at any time according to the service change requirement, meets the service change speed, improves the testing efficiency of big data, ensures the consistency of the data table structure of the service data and the data table structure extracted to the ODS layer, ensures the consistency of the incremental data of the service system and the incremental data of the upper layer interface of the multi-bin, improves the testing quality and has comprehensive service scene coverage; in addition, a complete data flow closed loop is formed by simulating and testing the data verification of all service data flows- > ODS layer- > DW layer- > ADS layer- > multi-bin upper interface layer- > application systems, and the test quality of big data is further ensured.
Example two
Fig. 2 is a structural diagram of a big data testing apparatus according to a second embodiment of the present invention.
In some embodiments, the big data testing apparatus 20 may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the big data testing apparatus 20 may be stored in a memory of the electronic device and executed by at least one processor to perform the functions of big data testing (described in detail in fig. 1).
In this embodiment, the big data testing apparatus 20 may be divided into a plurality of functional modules according to the functions performed by the big data testing apparatus. The functional module may include: the system comprises a first extraction module 201, a second extraction module 202, a use case generation module 203, a data test module 204, a data push module 205 and a verification test module 206. The module referred to herein is a series of computer program segments capable of being executed by at least one processor and capable of performing a fixed function and is stored in memory. In the present embodiment, the functions of the modules will be described in detail in the following embodiments.
The first extraction module 201 is configured to, when detecting that target service data is stored in a service system database, extract the target service data to a number bin ODS layer.
The business data produced by the business system is stored in a database of the business system, and the business data refers to the data which is produced by the business system in the actual production environment and is related to the business. For the convenience of the following description, the service data stored in the service system database is referred to as target service data.
The big data testing platform is in butt joint with one or more service systems so as to carry out full testing on the service data of the one or more service systems.
The big data test platform can actively detect whether the target service data is stored in the database of the service system in real time, and the service system can also actively report the target service data after storing the target service data in the database.
The big data testing platform can extract the target service data to the multi-bin ODS layer through the installed Jerner testing tool, and therefore testing is conducted.
The multiple bins of the big data testing platform can comprise 4 layers of architectures, namely an ODS layer, a DW layer, an ADS layer and an upper interface layer.
In an optional embodiment, the extracting, by the first extraction module 201, the target service data into a bin ODS layer includes:
acquiring a plurality of data fields of the target service data;
generating an ODS test script according to the plurality of data fields;
calling the ODS test script to obtain a binlog log of the target service data;
controlling the analysis progress of the binlog according to the identifier of the binlog log;
sending the analyzed data to a Kafka queue;
and extracting the data in the Kafka queue to the number bin ODS layer based on the corresponding relation configured in the number bin ODS layer in advance.
Due to the fact that business systems are various and databases used by the business systems are different, time and labor are consumed in the process of loading target business data to the multi-bin ODS layer. The ODS test script is generated according to a plurality of data fields of the target service data, the ODS test script can be used for automatically extracting data, and the ODS test script is suitable for various service systems and various databases of the service systems.
In addition, the analyzed identifier of the binlog log progress is stored in the Redis, and a tester can switch the analysis progress of the binlog by changing the corresponding numerical value in the Redis, so that the flexibility is improved. When the analysis progress of the binlog is too fast, the corresponding numerical value in the Redis can be reduced, so that the analysis progress of the binlog is reduced, when the analysis progress of the binlog is too slow, the corresponding numerical value in the Redis can be increased, the analysis progress of the binlog is improved, and therefore the target business data can be effectively prevented from being lost in the loading process, and the safety of loading the target business data is improved.
The second extracting module 202 is configured to extract the service data in the warehouse ODS layer to the warehouse DW layer when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the warehouse ODS layer.
After extracting the target service data to the multi-bin ODS layer, the big data testing platform firstly judges whether the data table structure of the target service data is consistent with the data table structure of the service data extracted to the multi-bin ODS layer. And extracting the service data of the multi-bin ODS layer to the multi-bin DW layer only when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the multi-bin ODS layer.
In an optional implementation, the second extraction module 202 is further configured to:
when the data table structure of the target service data is inconsistent with the data table structure of the service data extracted to the ODS layer of the bins, deleting the service data of the ODS layer of the bins;
deriving a table building statement of the target service data from a database of the service system;
executing the table building statement in the number-bin ODS layer such that the target service data is synchronized to the number-bin ODS layer.
The big data test platform analyzes the table building sentence to obtain the table name of the table to be built, the field name of the table header field, the field type and the field format, wherein the field type comprises a character type, a numerical value type and the like, and the field format is the format of the value corresponding to the field.
By exporting the table building statement of the target service data and executing the table building statement in the multi-bin ODS layer, the target service data can be quickly synchronized into the multi-bin ODS layer.
The use case generating module 203 is configured to generate a new successful test use case.
The test case set generally comprises successful test cases and failed test cases, and in order to improve the test quality of the data test, the more successful test cases are, the better is. The big data test platform can judge the number of successful test cases firstly, when the number is lower than a preset number threshold value, a new successful test case can be generated based on the successful test cases in the test case set firstly, the new generated test cases are combined into the test case set to obtain a new test case set, and therefore data test is conducted on business data on the basis of the new test case set on the multi-cabin ADS layer.
In an optional embodiment, the generating process of the new successful test case includes:
acquiring a successful test case set in an original test case set;
searching K nearest neighbor successful test cases of each successful test case in the successful test case set;
selecting a target successful test case from the successful test case set each time;
generating a random integer between 1 and K, and generating candidate test cases by using a test case calculation model according to the target successful test case and the random integer;
testing the candidate test case;
and responding to the instruction that the candidate test case passes the test, and determining the candidate test case as the new successful test case.
The Euclidean distance between every two successful test cases can be calculated, the Euclidean distances are sorted from small to large for any successful test case, and the successful test cases corresponding to the first K Euclidean distances in the sorting process are used as K neighbor successful test cases of any successful test case.
Random integers between 1 and K may be generated using a random number generation algorithm such that candidate test cases are generated using a test case computation model based on the generated random integers. In order to determine that the generated candidate test case is a successful test case, the candidate test case is tested first.
In an optional embodiment, the generating a candidate test case by using a test case calculation model according to the target successful test case and the random integer includes:
generating a case array based on K nearest neighbor successful test cases of the target successful test case;
acquiring an array index corresponding to the random integer in the case array and a target nearest neighbor successful test case corresponding to the array index;
calculating a difference factor between the target successful test case and the target nearest neighbor successful test case;
generating a test seed by using a test case calculation model;
and generating candidate test cases based on the target successful test case, the test seeds and the difference factor.
Based on the distance between the nearest neighbor successful test case and the target successful test case, the nearest neighbor successful test case with small distance is preferentially written into the case array according to the distance sequence, and the nearest neighbor successful test case with large distance is finally written into the case array. After the K nearest neighbor successful test cases are successfully written into the case array, the array indexes correspond to the nearest neighbor successful test cases one to one.
The test case computation model may be a rand function, generating a pseudo-random number between 0 and 1 as a test seed.
In an alternative embodiment, the test case calculation model may be represented by the following formula: ti + rand (0, 1) × tr-ti |, wherein r represents a random integer, tr represents a target nearest neighbor successful test case corresponding to an array index which is the same as the random integer, and ti represents a selected ith target successful test case.
The data testing module 204 is configured to perform data testing on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case.
And the big data test platform extracts the service data of the multi-bin ODS layer to the multi-bin DW layer through an ETL tool, and extracts the service data of the multi-bin DW layer to the multi-bin ADS layer through the ETL tool.
The data pushing module 205 is configured to respond to the instruction that the data test passes, and push the service data extracted from the ADS layer to the upper interface layer of the several bins to a target application system.
After the data test of the service data of the ADS layer of the plurality of bins passes, the ETL tool is used for extracting the service data of the ADS layer of the plurality of bins to an interface layer at the upper layer of the plurality of bins. The upper interface layer of the several bins corresponds to one or more application systems, such as BI, HRP, etc.
The big data test platform traverses the service data extracted to the upper layer interface layer of the storage bins, and notifies the upper layer interface layer of the storage bins to push the service data to a target application system after verifying the correctness of the service data of the upper layer interface of the storage bins.
The verification test module 206 is configured to perform a consistency verification test on the service data pushed to the target application system and the target service data to obtain a consistency verification test result.
The consistency check test result may include that the consistency check test passes and the consistency check test fails.
In an optional embodiment, the checking and testing module 206 performs a consistency checking and testing on the service data pushed to the target application system and the target service data, and obtaining a consistency checking and testing result includes:
acquiring a page ID for producing the target service data in the service system;
using SQL query statements to query the business data of the target application system to obtain an SQL query result;
acquiring the numerical value of each row in the SQL query result and a result ID corresponding to the numerical value of each row;
acquiring a page ID which is the same as the result ID of each row;
comparing whether the numerical value corresponding to the result ID of each row is consistent with the service data of the same row corresponding to the same page ID;
and when the numerical values corresponding to the result IDs of all the rows are consistent with the service data of all the rows corresponding to the same page ID, the consistency verification test result is that the consistency verification test is passed.
And when the numerical value corresponding to the result ID of any row is inconsistent with the service data of all rows corresponding to the same page ID, the consistency verification test result is that the consistency verification test fails.
It should be emphasized that, in order to further ensure the privacy and security of the target service data in the service system database and the service data of each layer in the bins, the service data of each layer in the bins may also be stored in the nodes of a block chain.
The big data testing device provided by the invention can execute full test at any time according to the service change requirement, meets the service change speed, improves the testing efficiency of big data, ensures the consistency of the data table structure of the service data and the data table structure extracted to the ODS layer, ensures the consistency of the incremental data of the service system and the incremental data of the upper layer interface of the multi-bin, improves the testing quality and has comprehensive service scene coverage; in addition, a complete data flow closed loop is formed by simulating and testing the data verification of all service data flows- > ODS layer- > DW layer- > ADS layer- > multi-bin upper interface layer- > application systems, and the test quality of big data is further ensured.
EXAMPLE III
The present embodiment provides a computer-readable storage medium, which stores thereon a computer program, which when executed by a processor implements the steps in the above-mentioned big data testing method embodiment, such as S11-S15 shown in fig. 1:
s11, when detecting that the target service data is stored in the service system database, extracting the target service data to a multi-bin ODS layer;
s12, when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the ODS layer of the bins, extracting the service data of the ODS layer of the bins to the DW layer of the bins;
s13, performing data test on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case;
s14, responding to the instruction that the data test passes, pushing the service data extracted from the ADS layer to the upper interface layer of the plurality of bins to a target application system;
and S15, performing consistency check test on the service data pushed to the target application system and the target service data to obtain a consistency check test result.
Alternatively, the computer program, when executed by the processor, implements the functions of the modules/units in the above-mentioned device embodiments, for example, the module 201 and 206 in fig. 2:
the first extraction module 201 is configured to, when detecting that target service data is stored in a service system database, extract the target service data to a warehouse ODS layer;
the second extracting module 202 is configured to extract the service data of the ODS layer of the several bins to the DW layer of the several bins when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the ODS layer of the several bins;
the case generating module 203 is configured to generate a new successful test case;
the data testing module 204 is configured to perform data testing on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case;
the data pushing module 205 is configured to respond to the instruction that the data test passes, and push the service data extracted from the ADS layer to the upper interface layer of the several bins to a target application system;
the verification test module 206 is configured to perform a consistency verification test on the service data pushed to the target application system and the target service data to obtain a consistency verification test result.
Example four
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention. In the preferred embodiment of the present invention, the electronic device 3 comprises a memory 31, at least one processor 32, at least one communication bus 33 and a transceiver 34.
It will be appreciated by those skilled in the art that the configuration of the electronic device shown in fig. 3 does not constitute a limitation of the embodiment of the present invention, and may be a bus-type configuration or a star-type configuration, and the electronic device 3 may include more or less other hardware or software than those shown, or a different arrangement of components.
In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculation and/or information processing according to instructions set or stored in advance, and the hardware thereof includes but is not limited to a microprocessor, an application specific integrated circuit, a programmable gate array, a digital processor, an embedded device, and the like. The electronic device 3 may also include a client device, which includes, but is not limited to, any electronic product that can interact with a client through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device, for example, a personal computer, a tablet computer, a smart phone, a digital camera, and the like.
It should be noted that the electronic device 3 is only an example, and other existing or future electronic products, such as those that can be adapted to the present invention, should also be included in the scope of the present invention, and are included herein by reference.
In some embodiments, the memory 31 has stored therein a computer program that, when executed by the at least one processor 32, implements all or part of the steps of the big data test method as described. The Memory 31 includes a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an electronically Erasable rewritable Read-Only Memory (Electrically-Erasable Programmable Read-Only Memory (EEPROM)), an optical Read-Only disk (CD-ROM) or other optical disk Memory, a magnetic disk Memory, a tape Memory, or any other medium readable by a computer capable of carrying or storing data.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
In some embodiments, the at least one processor 32 is a Control Unit (Control Unit) of the electronic device 3, connects various components of the electronic device 3 by various interfaces and lines, and executes various functions and processes data of the electronic device 3 by running or executing programs or modules stored in the memory 31 and calling data stored in the memory 31. For example, the at least one processor 32, when executing the computer program stored in the memory, implements all or a portion of the steps of the big data test method described in embodiments of the present invention; or to implement all or part of the functionality of the big data test apparatus. The at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips.
In some embodiments, the at least one communication bus 33 is arranged to enable connection communication between the memory 31 and the at least one processor 32 or the like.
Although not shown, the electronic device 3 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 32 through a power management device, so as to implement functions of managing charging, discharging, and power consumption through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 3 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
The integrated unit implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, an electronic device, or a network device) or a processor (processor) to execute parts of the methods according to the embodiments of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or that the singular does not exclude the plural. A plurality of units or means recited in the specification may also be implemented by one unit or means through software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A big data testing method, the method comprising:
when detecting that target service data are stored in a service system database, extracting the target service data to a plurality of ODS layers;
when the data table structure of the target service data is consistent with the data table structure of the service data extracted to the multi-bin ODS layer, extracting the service data of the multi-bin ODS layer to a multi-bin DW layer;
performing data test on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case;
responding to the instruction that the data test passes, and pushing the service data extracted from the ADS layer to the upper interface layer of the plurality of bins to a target application system;
and carrying out consistency check test on the service data pushed to the target application system and the target service data to obtain a consistency check test result.
2. The big data testing method of claim 1, wherein the extracting the target business data into a number of bins (ODS) layer comprises:
acquiring a plurality of data fields of the target service data;
generating an ODS test script according to the plurality of data fields;
calling the ODS test script to obtain a binlog log of the target service data;
controlling the analysis progress of the binlog according to the identifier of the binlog log;
sending the analyzed data to a Kafka queue;
and extracting the data in the Kafka queue to the number bin ODS layer based on the corresponding relation configured in the number bin ODS layer in advance.
3. The big data test method of claim 1, wherein the method further comprises:
when the data table structure of the target service data is inconsistent with the data table structure of the service data extracted to the ODS layer of the bins, deleting the service data of the ODS layer of the bins;
deriving a table building statement of the target service data from a database of the service system;
executing the table building statement in the number-bin ODS layer such that the target service data is synchronized to the number-bin ODS layer.
4. The big data testing method of claim 1, wherein the generation process of the new successful test case comprises:
acquiring a successful test case set in an original test case set;
searching K nearest neighbor successful test cases of each successful test case in the successful test case set;
selecting a target successful test case from the successful test case set each time;
generating a random integer between 1 and K, and generating candidate test cases by using a test case calculation model according to the target successful test case and the random integer;
testing the candidate test case;
and responding to the instruction that the candidate test case passes the test, and determining the candidate test case as the new successful test case.
5. The big data testing method of claim 4, wherein the generating candidate test cases by using a test case calculation model according to the target successful test case and the random integer comprises:
generating a case array based on K nearest neighbor successful test cases of the target successful test case;
acquiring an array index corresponding to the random integer in the case array and a target nearest neighbor successful test case corresponding to the array index;
calculating a difference factor between the target successful test case and the target nearest neighbor successful test case;
generating a test seed by using a test case calculation model;
and generating candidate test cases based on the target successful test case, the test seeds and the difference factor.
6. The big data testing method of claim 5, wherein the test case calculation model is: ti + rand (0, 1) × tr-ti |, wherein r represents a random integer, tr represents a target nearest neighbor successful test case corresponding to an array index which is the same as the random integer, and ti represents a selected ith target successful test case.
7. The big data testing method according to any one of claims 1 to 6, wherein the performing a consistency check test on the service data pushed to the target application system and the target service data to obtain a consistency check test result comprises:
acquiring a page ID for producing the target service data in the service system;
using SQL query statements to query the business data of the target application system to obtain an SQL query result;
acquiring the numerical value of each row in the SQL query result and a result ID corresponding to the numerical value of each row;
acquiring a page ID which is the same as the result ID of each row;
comparing whether the numerical value corresponding to the result ID of each row is consistent with the service data of the same row corresponding to the same page ID;
and when the numerical values corresponding to the result IDs of all the rows are consistent with the service data of all the rows corresponding to the same page ID, the consistency verification test result is that the consistency verification test is passed.
8. A big data test apparatus, the apparatus comprising:
the system comprises a first extraction module, a data processing module and a data processing module, wherein the first extraction module is used for extracting target service data to a multi-bin ODS layer when detecting that the target service data are stored in a service system database;
a second extraction module, configured to extract the service data in the ODS layer of the several bins to the DW layer when a data table structure of the target service data is consistent with a data table structure of the service data extracted to the ODS layer of the several bins;
the data testing module is used for carrying out data testing on the service data extracted from the multi-bin DW layer to the multi-bin ADS layer based on the generated new successful test case;
the data pushing module is used for responding to the instruction that the data test passes and pushing the service data extracted from the ADS layer of the plurality of bins to the interface layer of the upper layer of the plurality of bins to a target application system;
and the verification test module is used for performing consistency verification test on the service data pushed to the target application system and the target service data to obtain a consistency verification test result.
9. An electronic device, comprising a processor and a memory, wherein the processor is configured to implement the big data testing method according to any one of claims 1 to 7 when executing the computer program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the big data testing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210061230.3A CN114416845A (en) | 2022-01-19 | 2022-01-19 | Big data testing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210061230.3A CN114416845A (en) | 2022-01-19 | 2022-01-19 | Big data testing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114416845A true CN114416845A (en) | 2022-04-29 |
Family
ID=81275484
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210061230.3A Pending CN114416845A (en) | 2022-01-19 | 2022-01-19 | Big data testing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114416845A (en) |
-
2022
- 2022-01-19 CN CN202210061230.3A patent/CN114416845A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209131B (en) | Method and system for determining faults of heterogeneous system based on machine learning | |
| CN104252481B (en) | The dynamic check method and apparatus of master-slave database consistency | |
| CN110309071B (en) | Test code generation method and module, and test method and system | |
| US8151248B1 (en) | Method and system for software defect management | |
| CN102236672B (en) | A kind of data lead-in method and device | |
| CN105718371B (en) | A kind of regression testing method, apparatus and system | |
| CN109684847A (en) | Self-repairing method, device, equipment and the storage medium of script loophole | |
| CN105095059A (en) | Method and device for automated testing | |
| CN113946499A (en) | Micro-service link tracking and performance analysis method, system, equipment and application | |
| CN107193730A (en) | A kind of interface test method of automation | |
| CN112948275A (en) | Test data generation method, device, equipment and storage medium | |
| CN111931471A (en) | Form collection method and device, electronic equipment and storage medium | |
| CN111782532A (en) | Software fault positioning method and system based on network abnormal node analysis | |
| CN114419631A (en) | Network management virtual system based on RPA | |
| CN112486957B (en) | Database migration detection method, device, equipment and storage medium | |
| CN112527568A (en) | Data flow testing method and device, electronic equipment and storage medium | |
| CN114416845A (en) | Big data testing method and device, electronic equipment and storage medium | |
| CN115310011A (en) | Page display method and system and readable storage medium | |
| CN114201328A (en) | Fault processing method and device based on artificial intelligence, electronic equipment and medium | |
| CN113282496A (en) | Automatic interface test method, device, equipment and storage medium | |
| CN112527655A (en) | Software version quality abnormity detection method and device, electronic equipment and storage medium | |
| CN111767222A (en) | Data model verification method and device, electronic equipment and storage medium | |
| CN107832211A (en) | Automatization test system and method based on event triggering implementation process monitoring | |
| CN110716778A (en) | Application compatibility testing method, device and system | |
| CN112612702B (en) | Automatic testing method and device based on web |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |