CN113342696A - Unit testing method and device and storage medium - Google Patents
Unit testing method and device and storage medium Download PDFInfo
- Publication number
- CN113342696A CN113342696A CN202110790597.4A CN202110790597A CN113342696A CN 113342696 A CN113342696 A CN 113342696A CN 202110790597 A CN202110790597 A CN 202110790597A CN 113342696 A CN113342696 A CN 113342696A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- module
- data set
- spark
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a unit testing method and a device and a storage medium, wherein the device comprises the following steps: the unit test loading module is used for registering the interface realization of the computing environment building module and loading the interface realization to the computing environment building module according to the defined Spark unit test, and registering the interface realization of the unit test module and loading the interface realization to the unit test module; the computing environment construction module is used for generating an engine construction task and a data storage device construction task according to the interface realization of the computing environment construction module; constructing a Spark calculation engine by using an engine construction task, and constructing a task construction data storage device by using a data storage device; the unit testing module is used for generating unit testing tasks according to the interface realization of the unit testing module; loading the unit test task into a Spark calculation engine for execution, and outputting a unit test report; and the data storage device is used for storing test data related to the unit test.
Description
Technical Field
The present invention relates to the field of software testing, and in particular, to a unit testing method and apparatus, and a storage medium.
Background
In a large-scale data computing scene, the Apache Spark is a mainstream big data computing engine at present due to the superior computing performance and the extremely strong expression capacity of the Apache Spark. The conventional Spark data calculation for production mainly reads data of a data storage device through a Spark calculation engine, processes the data, and finally writes the data into the data storage device, so that the operation dependence of Spark data calculation is complex, the current Spark operation environment only supports submission of a complete Spark calculation task, and it is difficult to pertinently perform a test on a fine-grained logic unit in the calculation task, so that the conventional test method generally considers the accuracy of an overall calculation result, checks the final result data of Spark calculation, and mainly uses a Structured Query Language (SQL) supported by the data storage device to check the result data, which may cause a large test workload and a low operation efficiency in a test process.
Disclosure of Invention
In order to solve the foregoing technical problems, embodiments of the present invention are intended to provide a unit testing method and apparatus, and a storage medium, which can reduce the testing workload of Spark test and improve the operating efficiency of Spark test.
The technical scheme of the invention is realized as follows:
the embodiment of the application provides a unit testing device, the device includes: the system comprises a unit test loading module, a computing environment construction module and a unit test module;
the unit test loading module is used for registering and loading the interface realization of the computing environment building module to the computing environment building module according to the defined Spark unit test, and registering and loading the interface realization of the unit test module to the unit test module;
the computing environment construction module is used for generating an engine construction task and a data storage device construction task according to the interface realization of the computing environment construction module; constructing a Spark calculation engine by using the engine construction task, and constructing a task construction data storage device by using the data storage device;
the unit testing module is used for generating unit testing tasks according to the interface realization of the unit testing module; loading the unit test task into the Spark calculation engine for execution, and outputting a unit test report;
the data storage device is used for storing the test data related to the unit test.
The embodiment of the application provides a unit testing method, which is applied to the unit testing device and comprises the following steps:
constructing a Spark calculation engine and a data storage device according to the defined Spark unit test; the data storage device is configured to support read-write access by the Spark calculation engine;
importing the use case data into the data storage device;
executing a tested Spark unit in the Spark calculation engine based on the case data to obtain a unit execution result; and performing assertion inspection on the unit execution result to obtain a unit test report.
The embodiment of the application provides a storage medium, on which a computer program is stored, and the computer program is applied to a unit testing device, and when the computer program is executed by a processor, the computer program realizes the unit testing method.
The embodiment of the invention provides a unit testing method and device and a storage medium, wherein the device comprises the following steps: the system comprises a unit test loading module, a computing environment construction module and a unit test module; the unit test loading module is used for registering the interface realization of the computing environment building module and loading the interface realization to the computing environment building module according to the defined Spark unit test, and registering the interface realization of the unit test module and loading the interface realization to the unit test module; the computing environment construction module is used for generating an engine construction task and a data storage device construction task according to the interface realization of the computing environment construction module; constructing a Spark calculation engine by using an engine construction task, and constructing a task construction data storage device by using a data storage device; the unit testing module is used for generating unit testing tasks according to the interface realization of the unit testing module; loading the unit test task into a Spark calculation engine for execution, and outputting a unit test report; and the data storage device is used for storing test data related to the unit test. By adopting the device implementation scheme, a Spark calculation engine and a corresponding data storage device are constructed for Spark units based on a Spark framework and a deployment mode provided by a designed data storage device, case data in a single test scene are imported into a unit test device, tested Spark units are executed in a local environment, assertion and inspection are carried out on output results of the units, and then unit test can be completed, so that targeted operation test of fine-grained logic units in Spark calculation tasks is realized, and the operation efficiency in the test process is improved while the test workload is reduced.
Drawings
Fig. 1 is a first schematic structural diagram of a unit testing apparatus according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a unit testing apparatus according to an embodiment of the present disclosure;
fig. 3 is a system architecture diagram of an exemplary spare unit testing device provided in an embodiment of the present application;
FIG. 4 is a flow chart of a unit testing method according to an embodiment of the present application;
fig. 5 is a flowchart illustrating an exemplary process of importing case data according to an embodiment of the present application;
FIG. 6 is a flowchart illustrating an exemplary implementation process for checking two distributed data sets for equality according to an embodiment of the present application;
fig. 7 is a flowchart illustrating an exemplary method for determining whether data records are consistent according to an embodiment of the present application.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Example one
The embodiment of the present application provides a unit testing apparatus 1, as shown in fig. 1, the apparatus 1 includes: a unit test loading module 10, a computing environment construction module 11 and a unit test module 12;
the unit test loading module 10 is configured to register an interface implementation of the computing environment building module 11 and load the interface implementation to the computing environment building module 11 according to a defined Spark unit test, and register an interface implementation of the unit test module 11 and load the interface implementation to the unit test module 12;
the computing environment construction module 11 is configured to generate an engine construction task and a data storage device construction task according to the interface implementation of the computing environment construction module 11; constructing a Spark calculation engine 110 by using the engine construction task, and constructing a task construction data storage device 111 by using the data storage device;
the unit testing module 12 is configured to generate a unit testing task according to the interface implementation of the unit testing module 12; loading the unit test task into the Spark calculation engine 110 for execution, and outputting a unit test report;
the data storage device 111 is used for storing the test data related to the unit test.
The unit testing device provided by the embodiment of the application is used for realizing Spark unit testing.
In the embodiment of the application, the unit testing device inputs the Spark unit test defined by the interface implementation according to the predetermined interface definition into the unit test loading module, the unit test loading module is responsible for loading and running the whole Spark unit test, and registers and loads the Spark unit test-based interface implementation in the computing environment building module and the unit test module to the corresponding module.
It should be noted that the agreed interface definitions include an interface definition constructed by the computing environment, an interface definition imported by use case data, an interface definition executed by the unit, and an interface definition checked by the assertion.
In the embodiment of the application, a computing environment construction module realizes generation of a local environment task according to a Spark unit test at an interface of the module, wherein the local environment task comprises an engine construction task and a data storage device construction task, the computing environment construction module utilizes the engine construction task to construct a corresponding Spark computing engine locally, and the data storage device can be accessed by Spark reading and writing and has a local deployment solution.
It should be noted that the computing environment building module builds the local Spark calculation engine and the data storage device according to the definition of the computing environment in the interface implementation of the computing environment building module. The construction method is constructed through a Spark calculation engine and a local construction method of the corresponding data storage device.
It should be noted that the interface definition constructed by the computing environment is used for defining the interface constructed by the computing environment to implement the relevant configuration of the Spark computing engine and the data storage device, where the relevant configuration is composed of the startup parameters agreed by the Spark framework, the data storage device to be constructed, and the startup parameters of the data storage device to be constructed;
optionally, the type of the data storage device may be a Hive data warehouse, a Mysql database, a Hadoop database, or the like, which may be specifically selected according to an actual situation, and this embodiment of the present application is not specifically limited.
In the embodiment of the application, the test data stored in the data storage device comprises use case data and unit execution results; and the Saprk calculation engine is used for reading and writing case data and unit calculation output data from the data storage device in the process of executing the unit test task.
It can be understood that the unit test device supports data storage devices which are limited to Spark read-write access and have local deployment solutions, and can greatly meet the unit test of computing logic of Spark in various data storage devices.
It should be noted that the computing environment building module builds the local spare computing engine through the local boot mode agreed by the spare framework, and builds the local data storage device through the local building method agreed by the data storage device.
In the embodiment of the application, the unit testing module is implemented at an interface of the module according to Spark unit testing, a unit testing task is generated, then the unit testing task is loaded into a Spark calculation engine which is constructed before and executed, the Spark calculation engine performs Spark unit testing by accessing testing data related to the unit testing stored in the data storage device, and a unit testing report is obtained and output.
Alternatively, referring to fig. 2, the unit test module 12 includes: a use case data import module 120, a unit execution module 121 and a predicate check module 122;
the unit test loading module 10 is configured to register an interface implementation of the use case data import module 120 and load the interface implementation to the use case data import module 120, register an interface implementation of the unit execution module 121 and load the interface implementation to the unit execution module 121, and register an interface implementation of the assertion checking module 122 and load the interface implementation to the assertion checking module 122 according to the Spark unit test;
the use case data import module 120 is configured to generate a use case import task according to an interface of the use case data import module 120; and the case data corresponding to the unit test is imported into the data storage device 111 through the case import task;
the unit execution module 121 is configured to package the spare unit to be tested into a unit calculation task according to the interface implementation of the unit execution module 121; loading the unit calculation task into the Spark calculation engine 110 for execution to obtain a unit execution result;
the assertion verifying module 122 is configured to encapsulate an assertion verifying method for the unit execution result into an assertion verifying task according to an interface implementation of the assertion verifying module 122; and loads the assertion checking task into the Spark calculation engine 110 for execution, so as to obtain the unit test report.
In the embodiment of the application, the unit testing module is composed of three modules: the unit test loading module is used for registering the interface realization of the case data import module and loading the interface realization to the case data import module, registering the interface realization of the unit execution module and loading the interface realization to the unit execution module and registering the interface realization of the assertion verification module and loading the interface realization of the assertion verification module to the assertion verification module according to Spark unit test.
In the embodiment of the application, the use case data import module realizes the generation of the use case import task at the interface of the module according to the Spark unit test, and imports the use case data into the data storage device.
In the embodiment of the application, the use case data import module obtains the use case data according to the use case data source defined in the interface implementation of the use case data import module according to the Spark unit test.
The example data may be imported into the data storage device by writing the example data into the corresponding data storage device using Spark, or by a method provided in the data storage device.
The interface definition of use case data import is used for defining the data attribute of the use case data import module in the data storage device for realizing the use case data source and the use case data.
Optionally, the use-case data source is a data source that supports forms such as a Uniform Resource Locator (URL), a local file, and a Java Database connection (JDBC), and is specifically selected according to an actual situation, which is not specifically limited in the embodiment of the present application.
For example, when the data storage device is a Hive data warehouse, data attributes such as a table name, a table structure, a table serialization deserializer, a storage format and the like of the Hive table of the use case data are defined.
In the embodiment of the application, the unit execution module loads the tested Spark unit into the Spark calculation engine according to the Spark unit testing executable Spark calculation logic code in the interface implementation of the unit execution module, and the Spark calculation engine reads case data from the data storage device as the input of the unit calculation task to perform the unit testing, so as to obtain the unit execution result.
It should be noted that the interface definition executed by the unit is used to define the spare unit implementation interface to implement the spare unit under test.
In the embodiment of the present application, the unit execution result is written in the data storage device as a kind of test data related to the unit test.
In the embodiment of the application, the assertion verifying module packages an assertion verifying task according to an assertion verifying method defined by the Spark unit test in the interface implementation of the assertion verifying module for unit execution results, and loads the assertion verifying task into the Spark calculation engine for execution. And the Spark calculation engine reads the unit execution result from the data storage device and performs assertion check by using the assertion check task to obtain a unit test report.
In the embodiment of the application, the interface definition of the assertion verification is used for defining the assertion verification interface to realize an assertion verification method, and the assertion verification method built in the unit test device is used for setting assertion for the unit execution result and performing assertion verification.
Based on the above embodiments, the embodiments of the present application further provide a predicate checking method, as shown in table 1, the predicate checking method can be selected from the predicate checking methods to perform predicate checking on a unit execution result. And therefore, various Spark unit calculation scenes can be supported for assertion checking.
TABLE 1 assertion verification method
Based on the foregoing embodiments, an embodiment of the present application provides a Spark unit testing device, which can implement Spark unit testing according to interface definitions provided by each module by using Java, Scala, Python, and R languages under an agreed specification. Fig. 3 shows a system architecture diagram of the Spark unit testing apparatus, where the Spark unit testing apparatus is composed of a unit testing loading module, an intelligent local computing environment building module, a use case data importing module, a unit executing module, an assertion verifying module, a Spark computing engine, and a data storage device.
Specifically, interface implementation is completed according to interface definition agreed by the device to define a Spark unit test, the device is started, the custom unit test is input into a unit test loading module, the unit test loading module is responsible for loading and running the whole unit test, and the interface implementation of an intelligent local computing environment building module, a case data import module, a unit execution module and an assertion inspection module based on the unit test is registered and loaded into corresponding modules. The intelligent local computing environment building module is realized based on an interface of the intelligent local computing environment building module, generates a Spark engine building task and a data storage device building task, and respectively builds a Spark computing engine through the Spark engine building task and builds a data storage device through the data storage device. Together, the Spark compute engine and the data store comprise an intelligent local computing environment. The use case data import module generates a use case data import task according to an interface of the use case data import module, and the Spark calculation engine writes the use case data into the use case data based on the use case data import task; the unit execution module generates a unit calculation task according to an interface of the module, a Spark calculation engine reads case data from a data storage device and inputs the case data into the unit calculation task to obtain unit calculation output data, and the Spark calculation engine writes the unit calculation output data into the data storage device; and the Spark calculation engine reads unit calculation output data from the data storage device and inputs the unit calculation output data into the assertion checking task, and finally outputs a unit test report to the Spark unit.
It can be understood that the unit testing device can construct a Spark local computing environment and a corresponding data storage device for Spark units to be tested, import use case data in a single test scenario into the device as an input of a unit to be tested, execute the Spark units to be tested in the local environment, and perform assertion verification on output results of the Spark units to complete unit testing.
Example two
Based on the same inventive concept of the embodiments, the embodiments of the present application provide a unit testing method applied to the unit testing apparatus described in the first embodiment, as shown in fig. 4, the method includes:
s101, constructing a Spark calculation engine and a data storage device according to defined Spark unit tests; the data storage device is configured to support read and write access by the Spark compute engine.
And S102, importing the use case data into a data storage device.
S103, executing the Spark unit to be tested in a Spark calculation engine based on case data to obtain a unit execution result; and performing assertion inspection on the unit execution result to obtain a unit test report.
The unit testing method provided by the embodiment of the application is suitable for a scene of unit testing the Spark unit,
in the embodiment of the present application, the Spark unit test is defined by completing the interface implementation according to the interface definition agreed by the unit test apparatus, and the unit test apparatus is started. Registering a computing environment interface for realization, and constructing a Spark computing engine and a data storage device according to the definition of a computing environment in the realization, wherein the construction method is constructed by a Spark and a local construction method of the corresponding data storage device; the method comprises the following steps of realizing registration of a case data import interface, acquiring case data according to a case data source defined in the realization, and importing the case data into a data storage device, wherein the import mode can be written into a corresponding data storage device by using Spark or data import is carried out by a method of the data storage device; the registration unit executes the interface implementation, encapsulates the tested Spark unit defined in the implementation into a Spark task, and loads the Spark task into a Spark calculation engine for execution; registering assertion checking interface implementation, packaging the assertion checking method defined in the implementation for unit execution results into Spark tasks, and loading the Spark tasks into a Spark calculation engine for execution. And generating a unit test report according to the assertion check result.
It should be noted that, since the use case data import interface defines data attributes of a use case data source and data in the data storage device, which need to be implemented by the interface, the process of importing the use case data into the data storage device may specifically include: acquiring a use case data source according to the use case data import interface; loading use case data from a use case data source according to a reading mode of the use case data source; creating a use case data table in the data storage device according to the data attribute defined by the interface; and importing the use case data into a use case data table.
It should be noted that, after the use case data is loaded from the use case data source, the use case data may be temporarily stored in the memory, and after the use case data table is created, the use case data in the memory may be imported into the use case data table.
Taking the data storage device as a Hive data warehouse as an example, a specific use case data import process is shown in fig. 5, and includes:
1. and loading the use case data from the use case data source according to the interface, wherein the use case data source can be a local file, a URL file or a JDBC file.
2. And according to the configuration, creating a use case data Hive table in Hive.
3. And importing the use case data into the Hive table.
In the embodiment of the application, the unit execution result is a distributed data set supported by Spark, and the unit testing device further performs assertion check on the distributed data set by using a preset assertion method to obtain a unit testing report.
In an embodiment of the present application, the preset assertion checking method includes at least one of the following: checking whether a keyword exists in the distributed data set, checking whether the distributed data set is empty, checking the size of the distributed data set, checking whether a preset value exists in the distributed data set, checking whether a repeated item exists in the distributed data set, checking whether values in the value set exist in the distributed data set, checking whether the two distributed data sets are equal, checking whether a Boolean expression is true, and checking the line composition in the data frame (DataFrame) type distributed data set.
Optionally, the Spark-supported Distributed DataSet may be an abstract flexible Distributed Datasets (RDD), a data set (DataSet), and a DataFrame Distributed DataSet.
It should be noted that Spark calculation usually uses distributed data sets such as RDD, DataSet, and DataFrame to perform data calculation, where a data set usually consists of multiple data records, and a data record usually consists of multiple data fields, and the number of data fields of each data record should be kept consistent; based on this, an implementation process for verifying whether two distributed data sets are equal is proposed, where the two distributed data sets may be a first data set and a second data set, as shown in fig. 6 in particular, including:
1. the pattern of the first data set is compared with the pattern of the second data set.
2. If the pattern of the first data set and the pattern of the second data set are equal, the number of pieces of data of the first data set and the number of pieces of data of the second data set are compared.
3. If the number of the data pieces of the first data set is equal to the number of the data pieces of the second data set, the first data set is converted into a first local data set through a Spark gathering method, and the second data set is converted into a second local data set.
4. And respectively sequencing the first local data set and the second local data set, simultaneously traversing the sequenced first local data set and the sequenced second local data set, and judging whether the data records at the same position of the sequenced first local data set and the sequenced second local data set are consistent.
5. And if the data records at the same position of the sorted first local data set and the sorted second local data set are consistent, judging that the first data set and the second data set are equal.
6. And if the mode of the first data set is not equal to the mode of the second data set, or the number of the data pieces of the first data set is not equal to the number of the data pieces of the second data set, or the data records at the same position of the sequenced first local data set and the sequenced second local data set are inconsistent, judging that the first data set is not equal to the second data set.
Based on the foregoing embodiment, a method for determining whether data records are consistent is further provided, as shown in fig. 7, the method specifically includes:
1. and judging whether the lengths of the first data record and the second data record are consistent, wherein the first data record is a data record in the same position of the sorted first local data set and the sorted second local data set, and the second data record is a data record in the same position of the sorted second local data set and the sorted first local data set.
2. If the length of the first data record is consistent with that of the second data record, the first data record and the second data record are traversed at the same time, and whether the data field value of the first data record is equal to that of the second data record is compared.
3. If the data field value of the first data record is larger than the data field value of the second data record, judging that the first data record is larger than the second data record;
4. if the data field value of the first data record is smaller than the data field value of the second data record, judging that the first data record is smaller than the second data record;
5. if the data field value of the first data record is equal to the data field value of the second data record, the first data record is determined to be equal to the second data record.
It can be understood that the unit testing device can construct a Spark local computing environment and a corresponding data storage device for Spark units to be tested, import use case data in a single test scenario into the device as an input of a unit to be tested, execute the Spark units to be tested in the local environment, and perform assertion verification on output results of the Spark units to complete unit testing.
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a unit testing apparatus to implement the unit testing method as described above.
It can be understood that the unit testing device can construct a Spark local computing environment and a corresponding data storage device for Spark units to be tested, import use case data in a single test scenario into the device as an input of a unit to be tested, execute the Spark units to be tested in the local environment, and perform assertion verification on output results of the Spark units to complete unit testing.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (10)
1. A unit testing apparatus, the apparatus comprising: the system comprises a unit test loading module, a computing environment construction module and a unit test module;
the unit test loading module is used for registering and loading the interface realization of the computing environment building module to the computing environment building module according to the defined Spark unit test, and registering and loading the interface realization of the unit test module to the unit test module;
the computing environment construction module is used for generating an engine construction task and a data storage device construction task according to the interface realization of the computing environment construction module; constructing a Spark calculation engine by using the engine construction task, and constructing a task construction data storage device by using the data storage device;
the unit testing module is used for generating unit testing tasks according to the interface realization of the unit testing module; loading the unit test task into the Spark calculation engine for execution, and outputting a unit test report;
the data storage device is used for storing the test data related to the unit test.
2. The apparatus of claim 1, wherein the unit test module comprises: the system comprises a use case data import module, a unit execution module and an assertion checking module;
the unit test loading module is used for registering and loading the interface realization of the case data import module to the case data import module, registering and loading the interface realization of the unit execution module to the unit execution module, and registering and loading the interface realization of the assertion inspection module to the assertion inspection module according to the Spark unit test;
the use case data import module is used for generating a use case import task according to the interface of the use case data import module; importing the case data corresponding to the unit test into the data storage device through the case importing task;
the unit execution module is used for packaging the Spark unit to be tested into a unit calculation task according to the interface realization of the unit execution module; loading the unit calculation task into the Spark calculation engine for execution to obtain a unit execution result;
the assertion checking module is used for packaging an assertion checking method of the unit execution result into an assertion checking task according to the interface realization of the assertion checking module; and loading the assertion checking task into the Spark calculation engine for execution to obtain the unit test report.
3. The apparatus of claim 2, wherein the apparatus is further configured to define the spare cell test according to a contracted interface definition completion interface implementation; the appointed interface definition comprises an interface definition constructed by a computing environment, an interface definition imported by case data, an interface definition executed by a unit and an interface definition verified by assertion; wherein the content of the first and second substances,
the interface definition constructed by the computing environment is used for defining the relevant configuration of the Spark computing engine and the data storage device by the computing environment construction interface, wherein the relevant configuration is composed of starting parameters appointed by a Spark framework, a data storage device to be constructed and the starting parameters of the data storage device to be constructed;
the interface definition of the case data import is used for defining the data attribute of a case data source and case data in a data storage device realized by the case data import module;
the interface definition executed by the unit is used for defining the spare unit to realize the tested spare unit;
and the interface definition of the assertion checking is used for defining the assertion checking interface to realize the assertion checking method.
4. The apparatus of claim 1, wherein the data storage device is configured to support read-write access by the Spark calculation engine; the test data stored in the data storage device comprises case data and unit execution results;
the Spark calculation engine is configured to read and write the case data and the unit calculation output data from the data storage device in a process of executing the unit test task.
5. A unit testing method applied to the unit testing apparatus according to any one of claims 1 to 4, the method comprising:
constructing a Spark calculation engine and a data storage device according to the defined Spark unit test; the data storage device is configured to support read-write access by the Spark calculation engine;
importing the use case data into the data storage device;
executing a tested Spark unit in the Spark calculation engine based on the case data to obtain a unit execution result; and performing assertion inspection on the unit execution result to obtain a unit test report.
6. The method according to claim 5, wherein the importing the use case data into the data storage device comprises:
acquiring a use case data source;
loading use case data from the use case data source according to the reading mode of the use case data source;
creating a use case data table in the data storage device according to the data attribute defined by the interface;
and importing the use case data into the use case data table.
7. The method of claim 5, wherein the unit execution results in a Spark-supported distributed data set, the method further comprising:
and performing assertion inspection on the distributed data set by using a preset assertion inspection method to obtain the unit test report.
8. The method of claim 7, wherein the predetermined assertion verification method comprises at least one of:
checking whether a keyword exists in the distributed data set, checking whether the distributed data set is empty, checking the size of the distributed data set, checking whether a preset value exists in the distributed data set, checking whether a repeated item exists in the distributed data set, checking whether a value in a value set exists in the distributed data set, checking whether two distributed data sets are equal, checking whether a Boolean expression is true, and checking the line composition in the distributed data set of a data frame type.
9. The method of claim 8, wherein the two distributed data sets comprise a first data set and a second data set, and wherein verifying that the two distributed data sets are equal comprises:
comparing the pattern of the first data set with the pattern of the second data set;
comparing the number of pieces of data of the first data set with the number of pieces of data of the second data set if the pattern of the first data set and the pattern of the second data set are equal;
if the number of the data pieces of the first data set is equal to that of the data pieces of the second data set, converting the first data set into a first local data set and converting the second data set into a second local data set by a Spark gathering method;
respectively sequencing the first local data set and the second local data set, simultaneously traversing the sequenced first local data set and the sequenced second local data set, and judging whether the data records at the same position of the sequenced first local data set and the sequenced second local data set are consistent;
if the data records at the same position of the sorted first local data set and the sorted second local data set are consistent, judging that the first data set and the second data set are equal;
and if the mode of the first data set is not equal to the mode of the second data set, or the number of the data pieces of the first data set is not equal to the number of the data pieces of the second data set, or the data records at the same position of the sequenced first local data set and the sequenced second local data set are inconsistent, judging that the first data set is not equal to the second data set.
10. A storage medium having stored thereon a computer program for use in a unit testing apparatus, wherein the computer program when executed implements the method of any of claims 5 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110790597.4A CN113342696A (en) | 2021-07-13 | 2021-07-13 | Unit testing method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110790597.4A CN113342696A (en) | 2021-07-13 | 2021-07-13 | Unit testing method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113342696A true CN113342696A (en) | 2021-09-03 |
Family
ID=77479600
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110790597.4A Pending CN113342696A (en) | 2021-07-13 | 2021-07-13 | Unit testing method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113342696A (en) |
-
2021
- 2021-07-13 CN CN202110790597.4A patent/CN113342696A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107273286B (en) | Scene automatic test platform and method for task application | |
| US10635576B2 (en) | Branch coverage guided symbolic execution for hybrid fuzz testing of software binaries | |
| CN106557486A (en) | A kind of storage method and device of data | |
| JP2010524060A (en) | Data merging in distributed computing | |
| US10452628B2 (en) | Data analysis schema and method of use in parallel processing of check methods | |
| CN111966760B (en) | Test data generation method and device based on Hive data warehouse | |
| CN105446874A (en) | Method and device for detecting resource configuration file | |
| CN111597243A (en) | Data warehouse-based abstract data loading method and system | |
| CN111399911B (en) | Artificial intelligence development method and device based on multi-core heterogeneous computation | |
| Bajaber et al. | Benchmarking big data systems: A survey | |
| CN105824647A (en) | Form page generating method and device | |
| CN114461596A (en) | Multi-source data synchronization method, device and system based on Spark platform | |
| CN113342696A (en) | Unit testing method and device and storage medium | |
| CN112464636B (en) | Constraint file comparison method and device, electronic equipment and storage medium | |
| Fischer et al. | Large-Scale Benchmarking | |
| CN114461712A (en) | Method and device for importing and exporting multi-source heterogeneous data source and graph database | |
| CN113504904A (en) | User-defined function implementation method and device, computer equipment and storage medium | |
| CN117033497B (en) | Method for updating and reading data on data lake and related equipment | |
| CN108228604B (en) | Model construction method based on memory object, information query method and device | |
| CN116501650B (en) | Method, device and medium for executing carrier rocket measurement and control flow engine | |
| US11010387B2 (en) | Join operation and interface for wildcards | |
| CN116361288A (en) | Index recommendation method, device, equipment and storage medium | |
| CN117273112A (en) | Deep learning framework adjustment method, device, equipment, medium and program product | |
| US20240134888A1 (en) | User interface for providing an entity relationship diagram | |
| CN117707937A (en) | Unit test method, unit test device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |