CN117194265A - Automatic industry template testing method and device based on data warehouse hive - Google Patents
Automatic industry template testing method and device based on data warehouse hive Download PDFInfo
- Publication number
- CN117194265A CN117194265A CN202311234635.3A CN202311234635A CN117194265A CN 117194265 A CN117194265 A CN 117194265A CN 202311234635 A CN202311234635 A CN 202311234635A CN 117194265 A CN117194265 A CN 117194265A
- Authority
- CN
- China
- Prior art keywords
- data
- industry
- template
- industry template
- data warehouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 60
- 238000007405 data analysis Methods 0.000 claims abstract description 24
- 238000004458 analytical method Methods 0.000 claims abstract description 17
- 238000010998 test method Methods 0.000 claims abstract description 7
- 238000003860 storage Methods 0.000 claims description 17
- 238000005457 optimization Methods 0.000 claims description 6
- 238000011161 development Methods 0.000 abstract description 2
- 238000000034 method Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 239000008186 active pharmaceutical agent Substances 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000008676 import Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000007651 self-proliferation Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The application discloses an automatic test method and device for an industry template based on a data warehouse hive, and relates to the technical field of automatic test; importing the generated data into a data warehouse hive, and analyzing the data through the existing industry data analysis service; and generating a test report result based on analysis of the data by the industry data analysis service. The application supports the custom template, can effectively improve the testing efficiency, supports the importing and exporting of different data source data, and is convenient for testing personnel and service development personnel to optimize platform parameters and task parameters.
Description
Technical Field
The application relates to the technical field of automatic testing, in particular to an industry template automatic testing method and device based on a data warehouse hive.
Background
Only a real business scenario, namely commodity sales, is used by a standard TPC-DS to test the system performance. Query performance and data maintenance performance in single user mode and multi-user mode are achieved given hardware, operating system, and data processing system configurations. hive is a Hadoop (a distributed system infrastructure) based data warehouse tool.
However, the existing test method has the following problems: 1. the fact table and the dimension table are fixed, and a custom template cannot be supported, and the structure and the fields of the custom generation table cannot be supported; 2. the TPC-DS test is repeated, and in fact, some industries with clear relative service do not need to be covered with the sql99 test, so that manpower and time are wasted, and for the test of some industries, only some special services of the industries need to be covered; 3. TPC-DS does not support fast large-scale concurrent generation of imported data.
Disclosure of Invention
Aiming at the defects in the prior art, the application aims to provide the industrial template automatic testing method and device based on the data warehouse hive, which support the custom template, effectively improve the testing efficiency, support the importing and exporting of different data source data, and facilitate the optimization of platform parameters and task parameters by testing personnel and service development personnel.
In order to achieve the above purpose, the application provides an industry template automatic test method based on a data warehouse hive, which specifically comprises the following steps:
acquiring an industry template to be subjected to automatic test, and generating data based on the acquired industry template;
importing the generated data into a data warehouse hive, and analyzing the data through the existing industry data analysis service;
and generating a test report result based on analysis of the data by the industry data analysis service.
On the basis of the technical scheme, the method for acquiring the industry template to be subjected to the automatic test comprises the following specific steps:
loading the existing industry template, or modifying the existing industry template, or customizing the industry template, thereby obtaining the industry template to be automatically tested.
On the basis of the technical proposal, the method comprises the following steps,
the industry template comprises a custom table structure and a table field type;
the table field type includes long, int, string;
and the tables in the industry templates are associated according to the id key value.
On the basis of the technical scheme, the data generation based on the acquired industry template comprises the following specific steps:
and generating data according to the custom table structure and the table field type of the industry template.
Based on the above technical solution, the importing the generated data into the data warehouse hive specifically includes:
the generated data is imported into a data warehouse hive, kafka or other storage middleware.
On the basis of the technical scheme, the test report result comprises a task execution flow, an optimization suggestion and execution time.
The application provides an industry template automatic testing device based on a data warehouse hive, which comprises:
the acquisition module is used for acquiring an industry template to be subjected to automatic test and generating data based on the acquired industry template;
the analysis module is used for importing the generated data into a data warehouse hive and analyzing the data through the existing industry data analysis business;
and the generating module is used for generating a test report result based on the analysis of the data by the industry data analysis service.
On the basis of the technical scheme, the method for acquiring the industry template to be subjected to the automatic test comprises the following specific steps:
loading the existing industry template, or modifying the existing industry template, or customizing the industry template, thereby obtaining the industry template to be automatically tested.
On the basis of the technical proposal, the method comprises the following steps,
the industry template comprises a custom table structure and a table field type;
the table field type includes long, int, string;
and the tables in the industry templates are associated according to the id key value.
On the basis of the technical scheme, the data generation based on the acquired industry template comprises the following specific steps:
and generating data according to the custom table structure and the table field type of the industry template.
Compared with the prior art, the application has the advantages that: the application supports the self-defined template, can effectively improve the testing efficiency, supports the importing and exporting of different data source data, and is convenient for testers and service developers to optimize platform parameters and task parameters.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an automated testing method for an industry template based on a data warehouse hive in an embodiment of the application.
Detailed Description
The embodiment of the application provides an automatic test method for an industry template based on a data warehouse hive, which is characterized in that the industry template to be automatically tested is obtained, data is generated based on the obtained industry template, the generated data is imported into the data warehouse hive, the data is analyzed through the existing industry data analysis service, and finally, a test report result is generated based on the analysis of the data by the industry data analysis service. The embodiment of the application correspondingly provides an industry template automatic testing device based on the data warehouse hive.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application.
Referring to fig. 1, the method for automatically testing an industry template based on a data warehouse hive provided by the embodiment of the application specifically comprises the following steps:
s1: acquiring an industry template to be subjected to automatic test, and generating data based on the acquired industry template;
in the application, the industry template to be automatically tested is obtained, and the specific steps include: loading the existing industry template, or modifying the existing industry template, or customizing the industry template, thereby obtaining the industry template to be automatically tested. That is, the user can load an existing industry template, modify the existing industry template, or customize the industry template, thereby obtaining an industry template to be automatically tested.
In actual operation, industry templates may be provided by Industry Template module, industry Template module providing special templates for different industries, such as: in the financial industry, the templates are iterated continuously, and the custom templates, custom table fields and custom field types are supported.
In the application, the industry template comprises a custom table structure and a table field type; the table field type includes long, int, string; the tables in the industry template are related according to an id (identity number) key value.
Namely, the table field type comprises common long, int, string and other data types, the tables can be associated according to id key values, and ids can be randomly self-increased or self-increased.
In the application, the data generation is carried out based on the acquired industry template, and the specific steps comprise: and generating data according to the custom table structure and the table field type of the industry template.
The industry template is a dimension table and a fact table constructed according to the business of the industry, each table has a corresponding metadata file comprising basic user information, id, name and the like, when the data generator loads the template, corresponding data is generated according to the type of each field according to the metadata information, for example, if id is generated according to self-proliferation, the user id generates data according to self-proliferation, if name definition is 10 English combinations, the data is generated randomly and repeatedly, and different field rule generation is supported.
In actual work, data generation can be performed through a Data Generator module, and the Data Generator module generates Data with different Data amounts according to table fields and table field types of a custom template and supports importing different Data sources.
S2: importing the generated data into a data warehouse hive, and analyzing the data through the existing industry data analysis service;
in the application, the generated data is imported into a data warehouse hive, which concretely comprises the following steps: the generated data is imported into a data warehouse hive, kafka (a high throughput distributed publish-subscribe messaging system) or other storage middleware.
I.e., after the data is generated, the import of the data into the data warehouse hive, kafka, or other storage middleware may be supported.
In actual work, substitution and analysis of Data may be performed by a Test Program module that supports execution of relevant Test scripts and generation of relevant Test reports after a Data producer Data Generator generates relevant Data files and imports them into a Data warehouse hive.
S3: and generating a test report result based on analysis of the data by the industry data analysis service.
In the application, the test report result comprises the task execution flow, the optimization suggestion and the execution time, so that the whole business process is more intuitively and effectively analyzed, and the platform parameters and the task parameters are conveniently optimized by testers and business developers.
The application relates to an industry template automatic test method based on a data warehouse hive, which comprises the following steps: 1. the self-defined templates are supported, including industry templates which are accumulated, the self-defined field types are supported, different random key values and unique key values can be generated, mass data quantity measurement data (generated by utilizing hadoop mr tasks or spark tasks) can be generated according to requirements; 2. the industry templates are used for simulating data forms of different industries according to experiences of the different industries, so that the problem of industry data pain points can be solved in a targeted manner, the adjustment of the platform is guaranteed before production is on line, and production is guaranteed; 3. support different data source data importation and exportation, and support automatic testing and automatic report generation.
According to the automatic test method for the industry template based on the data warehouse hive, the industry template to be automatically tested is obtained, data is generated based on the obtained industry template, the generated data is imported into the data warehouse hive, the data is analyzed through the existing industry data analysis service, and finally, a test report result is generated based on the analysis of the data by the industry data analysis service.
In a possible implementation manner, the embodiment of the present application further provides a non-transitory computer readable storage medium, where the readable storage medium is located in a PLC (Programmable Logic Controller ) controller, and a computer program is stored on the readable storage medium, and when the program is executed by a processor, the steps of the industry template automatic testing device based on the data warehouse hive are implemented as follows:
acquiring an industry template to be subjected to automatic test, and generating data based on the acquired industry template;
importing the generated data into a data warehouse hive, and analyzing the data through the existing industry data analysis service;
and generating a test report result based on analysis of the data by the industry data analysis service.
In the application, the industry template to be automatically tested is obtained, and the specific steps include:
loading the existing industry template, or modifying the existing industry template, or customizing the industry template, thereby obtaining the industry template to be automatically tested.
In the application, the industry template comprises a custom table structure and a table field type;
the table field type includes long, int, string;
and the tables in the industry templates are associated according to the id key value.
In the application, the data generation is carried out based on the acquired industry template, and the specific steps comprise:
and generating data according to the custom table structure and the table field type of the industry template.
In the application, the generated data is imported into a data warehouse hive, which concretely comprises the following steps:
the generated data is imported into a data warehouse hive, kafka or other storage middleware.
In the application, the test report result comprises a task execution flow, an optimization suggestion and an execution time.
According to the application, the industry template to be automatically tested is obtained, data generation is carried out based on the obtained industry template, the generated data is imported into the data warehouse hive, the data is analyzed through the existing industry data analysis service, and finally the test report result is generated based on the analysis of the data by the industry data analysis service.
The storage media may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The embodiment of the application provides an industrial template automatic testing device based on a data warehouse hive, which comprises an acquisition module, an analysis module and a generation module.
The acquisition module is used for acquiring an industry template to be subjected to automatic test and generating data based on the acquired industry template; the analysis module is used for importing the generated data into a data warehouse hive and analyzing the data through the existing industry data analysis business; the generating module is used for generating a test report result based on analysis of the data by the industry data analysis service.
In the application, the industry template to be automatically tested is obtained, and the specific steps include:
loading the existing industry template, or modifying the existing industry template, or customizing the industry template, thereby obtaining the industry template to be automatically tested.
In the application, the industry template comprises a custom table structure and a table field type;
the table field type includes long, int, string;
and the tables in the industry templates are associated according to the id key value.
In the application, the data generation is carried out based on the acquired industry template, and the specific steps comprise:
and generating data according to the custom table structure and the table field type of the industry template.
In the application, the generated data is imported into a data warehouse hive, which concretely comprises the following steps:
the generated data is imported into a data warehouse hive, kafka or other storage middleware.
In the application, the test report result comprises a task execution flow, an optimization suggestion and an execution time.
According to the automatic test device for the industry template based on the data warehouse hive, the industry template to be automatically tested is obtained, data is generated based on the obtained industry template, the generated data is imported into the data warehouse hive, the data is analyzed through the existing industry data analysis service, and finally, a test report result is generated based on the analysis of the data by the industry data analysis service.
The foregoing is only a specific embodiment of the application to enable those skilled in the art to understand or practice the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Claims (10)
1. An industry template automatic test method based on a data warehouse hive is characterized by comprising the following steps:
acquiring an industry template to be subjected to automatic test, and generating data based on the acquired industry template;
importing the generated data into a data warehouse hive, and analyzing the data through the existing industry data analysis service;
and generating a test report result based on analysis of the data by the industry data analysis service.
2. The automated data warehouse hive-based industry template testing method according to claim 1, wherein the step of obtaining the industry template to be automatically tested comprises the following specific steps:
loading the existing industry template, or modifying the existing industry template, or customizing the industry template, thereby obtaining the industry template to be automatically tested.
3. An automated testing method for industry templates based on a data warehouse hive as claimed in claim 2, wherein:
the industry template comprises a custom table structure and a table field type;
the table field type includes long, int, string;
and the tables in the industry templates are associated according to the id key value.
4. The automated data warehouse hive-based industry template testing method as claimed in claim 3, wherein the step of generating data based on the obtained industry template comprises the following specific steps:
and generating data according to the custom table structure and the table field type of the industry template.
5. The automated testing method of industry templates based on data warehouse hive according to claim 1, wherein the importing the generated data into the data warehouse hive specifically comprises:
the generated data is imported into a data warehouse hive, kafka or other storage middleware.
6. An automated testing method for industry templates based on a data warehouse hive as claimed in claim 1, wherein: the test report results include task execution flow, optimization suggestions, and execution time.
7. An automated testing device for an industry template based on a data warehouse hive, comprising:
the acquisition module is used for acquiring an industry template to be subjected to automatic test and generating data based on the acquired industry template;
the analysis module is used for importing the generated data into a data warehouse hive and analyzing the data through the existing industry data analysis business;
and the generating module is used for generating a test report result based on the analysis of the data by the industry data analysis service.
8. The automated data warehouse hive based industry template testing apparatus of claim 7, wherein the step of obtaining the industry template to be automatically tested comprises the steps of:
loading the existing industry template, or modifying the existing industry template, or customizing the industry template, thereby obtaining the industry template to be automatically tested.
9. An automated data warehouse hive based industry template testing device as claimed in claim 8, wherein:
the industry template comprises a custom table structure and a table field type;
the table field type includes long, int, string;
and the tables in the industry templates are associated according to the id key value.
10. An automated data warehouse hive based industry template testing device as claimed in claim 9, wherein the data generation based on the acquired industry template comprises the following specific steps:
and generating data according to the custom table structure and the table field type of the industry template.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311234635.3A CN117194265A (en) | 2023-09-21 | 2023-09-21 | Automatic industry template testing method and device based on data warehouse hive |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311234635.3A CN117194265A (en) | 2023-09-21 | 2023-09-21 | Automatic industry template testing method and device based on data warehouse hive |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117194265A true CN117194265A (en) | 2023-12-08 |
Family
ID=88990444
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311234635.3A Pending CN117194265A (en) | 2023-09-21 | 2023-09-21 | Automatic industry template testing method and device based on data warehouse hive |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117194265A (en) |
-
2023
- 2023-09-21 CN CN202311234635.3A patent/CN117194265A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107943463B (en) | Interactive mode automation big data analysis application development system | |
| CN107273286B (en) | Scene automatic test platform and method for task application | |
| CN108614770B (en) | Automatic test assertion method, device, storage medium and equipment | |
| CN106294673B (en) | Method and system for analyzing log data in real time by user-defined rule | |
| CN106777101B (en) | Data processing engine | |
| CN108345532A (en) | A kind of automatic test cases generation method and device | |
| CN113326247B (en) | Cloud data migration method and device and electronic equipment | |
| JP2012123789A (en) | Parallel work load simulation for application performance test | |
| CN112306861A (en) | Unittest and Jenkins tool-based interface automatic testing system and method | |
| CN114115904B (en) | Information processing method, device, server and storage medium | |
| CN111966760B (en) | Test data generation method and device based on Hive data warehouse | |
| CN112435072A (en) | Model creating method and device, electronic equipment and storage medium | |
| CN111221727A (en) | Test method, test device, electronic equipment and computer readable medium | |
| CN113419789A (en) | Method and device for generating data model script | |
| CN113297081B (en) | Execution method and device of continuous integrated pipeline | |
| Rezig et al. | Debugging large-scale data science pipelines using dagger | |
| CN104765787B (en) | Salt file association relationship analysis method and system in software systems | |
| Sun et al. | A web testing platform based on hybrid automated testing framework | |
| CN113032256A (en) | Automatic test method, device, computer system and readable storage medium | |
| CN117235527A (en) | End-to-end containerized big data model construction method, device, equipment and medium | |
| CN116431668A (en) | Metadata acquisition-based data blood-edge analysis method and device and electronic equipment | |
| CN117194265A (en) | Automatic industry template testing method and device based on data warehouse hive | |
| CN116402325A (en) | Automatic business process processing method and device | |
| CN115757174A (en) | Database difference detection method and device | |
| CN114385155A (en) | vue project visualization tool generation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |