CN110825641B - Micro-service application test system based on simulation data generator - Google Patents
Micro-service application test system based on simulation data generator Download PDFInfo
- Publication number
- CN110825641B CN110825641B CN201911090197.1A CN201911090197A CN110825641B CN 110825641 B CN110825641 B CN 110825641B CN 201911090197 A CN201911090197 A CN 201911090197A CN 110825641 B CN110825641 B CN 110825641B
- Authority
- CN
- China
- Prior art keywords
- service
- interface
- data
- information
- mock
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 75
- 238000004088 simulation Methods 0.000 title claims abstract description 24
- 238000013507 mapping Methods 0.000 claims abstract description 52
- 238000013461 design Methods 0.000 claims abstract description 14
- 238000000034 method Methods 0.000 claims description 25
- 230000004044 response Effects 0.000 claims description 19
- 238000005516 engineering process Methods 0.000 claims description 17
- 238000010586 diagram Methods 0.000 claims description 14
- 238000012423 maintenance Methods 0.000 claims description 13
- 230000003993 interaction Effects 0.000 claims description 9
- 230000010354 integration Effects 0.000 claims description 3
- 238000013480 data collection Methods 0.000 description 22
- 238000007405 data analysis Methods 0.000 description 20
- 238000007726 management method Methods 0.000 description 15
- 238000012545 processing Methods 0.000 description 15
- 238000007781 pre-processing Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 12
- 238000004458 analytical method Methods 0.000 description 9
- 238000013500 data storage Methods 0.000 description 9
- 238000011161 development Methods 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 6
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000012163 sequencing technique Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000003672 processing method Methods 0.000 description 3
- 230000003139 buffering effect Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012827 research and development Methods 0.000 description 2
- 230000007474 system interaction Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a micro-service application testing tool set based on a simulated data generator, which comprises a display layer, an application layer and a data layer, wherein the application layer comprises service management, interface management, URL mapping, MOCK service and file generation, and the data layer comprises a file storage system and a mysql database; mysql database: the mysql database is used for storing service information and interface information base, and contains various information and characteristics of services and interfaces; a file storage system: the file storage system is used for storing mapping and __ file files of the interface; and (3) service management: the design of the scheme is used for managing service information, adding services, maintaining services, deleting micro-services and the like, realizes the functions of service and data simulation, and solves the problem of dependence on external services and data in unit test and integrated test.
Description
Technical Field
The invention relates to the technical field of micro-service application testing, in particular to a micro-service application testing system based on a simulation data generator.
Background
Micro-services are an emerging software architecture that splits a large single application and service into tens of supporting micro-services. A micro-service strategy may make the work easier and it may extend individual components rather than the entire application stack to meet service level agreements.
One of the greatest advantages of microservice applications is that they tend to utilize computing resources more efficiently than traditional applications. As they handle the functional bottleneck problem by expanding the components. In this way, the developer need only deploy computing resources for additional components, rather than deploying a completely new iteration of the complete application. The end result is that more resources are available for other tasks.
The application and implementation of the micro-service application testing technology in research and development and test operation can improve the testing convenience and the testing efficiency. Firstly, a micro-service application test tool set based on a simulation data generator (Mock) is researched, service and data simulation functions are realized, and the problem of dependence on external services and data in unit test and integrated test is solved.
Disclosure of Invention
The invention aims to provide a micro-service application testing system based on a simulation data generator, which aims to solve the problems that in the background art, in order to develop and realize the application and the test operation of a micro-service application testing technology, a micro-service application testing tool set based on a simulation data generator (Mock) needs to be developed to realize the service and data simulation functions.
In order to achieve the above purpose, the present invention provides the following technical solutions: the micro service application test system based on the simulation data generator comprises a presentation layer, an application layer and a data layer, and is characterized in that: the application layer comprises service management, interface management, URL mapping, MOCK service and file generation, and the data layer comprises a file storage system and a mysql database;
creating __ files and maps folders on a Mock server, wherein __ files are used for storing uploaded, downloaded or recorded files and used for receiving and returning interface request file data; the maps are used for storing service interface information data, including mapping information, request and response information of url of the interface;
typical service interface information file mapping. Json content is as follows:
when calling/api/testdetail interfaces in service unit testing and integration testing, the Mock server searches mapping files corresponding to the api/testdetail according to interface routing url mapping relation, and returns the content of testdetail.json, which is a response file to be built in __ files;
the Mock test platform also supports returning response content directly in response structure
The Mock test platform is responsible for managing files of maps and __, managing service requests and returning request result content files;
the micro-service Mock testing platform based on the simulation data generation technology is a micro-service application testing tool based on the simulation data generation technology, which is realized by integrating related key technologies and performing further optimized development on the basis of a micro-service application testing system and a micro-service application key technical scheme;
after service information and interface basic information are transmitted into a system, the service information and the interface basic information are stored in a database, the system generates mapping.json and __ file files of corresponding interfaces in a file generation mode, and the mapping.json and __ file files are stored in a file storage system;
after a user sends an interface simulation request, the address mapping routing system is called through a dock service to locate corresponding interface mapping and __ file files, interface return information is processed, and dock result data is finally returned;
the method comprises the steps that a dock test platform is divided into a dock foreground and a dock background according to design thought and system architecture diagrams, the dock foreground is mainly responsible for information maintenance of services and maintenance of service interface information, and the dock background is responsible for constructing dock API and API route mapping functions;
the method comprises the following steps:
(1) adding service information to be simulated by adopting a Mock technology in a Mock foreground;
(2) adding Mock interface information in the service, wherein the Mock interface information comprises the name, address, request and response information of an interface; (3) after the Mock interface information is stored, a Mock API is automatically constructed in a Mock background, an interface route is automatically added, and the requirements of simulating interface requests and returning interface data are met;
(4) calling an API interface outside a Mock test platform in a unit test script of a tested server or front end, sending interface request data, and returning result data of the Mock API interface by a Mock background;
(5) checking result data, and ending the test;
introducing a microservice application data collection and analysis tool from five aspects of system design thought, system implementation principle, system architecture design, system data architecture and system technology architecture;
1) System design concept
The micro-service application data collection and analysis tool is developed to monitor the micro-service in real time, collect service operation log data information, analyze the micro-service system according to the collected data information, and realize continuous optimization for the micro-service application through data feedback;
the micro service is a distributed architecture, the calling relation logic among the services is complex, the characteristics determine that the log data analysis of the micro service application needs to be carried out from two aspects, on one hand, the daily operation log of the service comprises a service operation log, a database operation log and a system fault error log; on the other hand, the service call chain tracks log information, and the service call chain is checked through analyzing the service call chain, so that the time consumption of the service call chain and the service bottleneck are checked;
the service running log and the service call chain log are the contents which are focused and analyzed by the micro-service application data collection and analysis tool;
the micro-service application data collection and analysis tool should collect service running logs and service call chain logs distributed on different servers, and establish a system crash mechanism for preventing the log from being in a peak period;
2) Principle of system implementation
(1) Distributed architecture log collection principle
In a distributed micro-service application system, logs are stored on different servers in a scattered way, and collecting log data from different services is the basic work of the data collection tool, in a micro-service architecture, a log back component is generally used for recording service logs by default, so an Appender applicator of a data buffer queue component kafka can be added in a log back configuration to realize the collection of log data;
after the related configuration is finished, related log data is automatically sent to a data buffer area in the running process of the micro-service, and the data can be obtained from the data buffer area and processed by data preprocessing;
(2) service invocation chain tracking principle
In the micro-service framework, a request initiated by a client is called by a plurality of different service nodes in a back-end system to cooperatively generate a final request result, each front-end request forms a complex distributed service call link, and high delay or errors of any ring in the link can cause the final failure of the whole request, so a distributed service tracking mechanism is required to be introduced to realize service tracking under the micro-service framework;
in the micro-service architecture, each service request generates a global unique traceID, and the entire call chain can be connected in series through the traceID, that is to say, when the distributed system internally circulates, the system needs to always keep transmitting the unique traceID until the request returns;
when a request arrives at each service component, its start, concrete procedure and end are marked by Span ID, it must have two nodes for each Span to start and end, and the Span's time delay is counted by recording the time stamps of the start Span and end Span;
in the whole calling process, each request is transmitted with a traceID and a Span ID, each service records the Span ID attached to the request as a PSANID and generates a Span ID of the service, and a Span without the PSANID is a root Span and can be regarded as a calling chain entry, so that all calling records are only required to be searched according to the traceID for checking a certain complete call, and then the whole calling father-son relationship is organized through the PSpanID and the Span ID;
3) System architecture design
The micro-service application data collection and analysis tool set is based on a micro-service application test system and a micro-service application key test technical scheme, integrates related technologies and performs further optimized development to realize a micro-service application data collection and analysis tool, and the tool is divided into a data layer, an application layer and a presentation layer;
the data collection layer mainly has the functions of service log data acquisition and data storage, and mainly comprises service log data collection, data caching, data preprocessing and data storage, the application layer further processes the preprocessed log data, and the application layer comprises statistics, analysis, sequencing and inquiry, and the display layer is used for visually displaying service log data analysis results and user interaction;
and (3) data acquisition: the data acquisition is used for collecting service running logs, service call logs and database running log information, and log information is obtained by using log back and slot technologies and is sent to the data buffer layer;
data buffering: the data buffer layer is mainly used for preventing the problem of system breakdown caused by overlarge instantaneous log data quantity, and can play a role in flow peak elimination as a front buffer to offset the unmatched problem of log flow peak and consumption;
data preprocessing problem: the number of the logs collected by default is huge, the data can be subjected to preliminary processing, and data processing rules including filtering, conversion and deformation operations are formulated;
and (3) data storage: storing the preprocessed log data into a database, wherein the log data comprises a service running log and a service calling link log; storing data by utilizing a Hadoop HDFS and Mysql database;
log processing: further processing is carried out on the preprocessed log information, including statistics, sequencing, inquiry and analysis functions;
system interaction and visualization platform: the method is mainly used for user interaction and data display;
4) System data architecture
The data buffer area is used for storing original micro service log data which is not subjected to data preprocessing, the system deletes corresponding log data in the data buffer area after the data preprocessing, and the preprocessed data is stored in the database;
the data storage library comprises micro service running logs, database running logs and service call chains for tracking log contents, and the data can show the final analysis result to a user after passing through the log processing module;
5) System technical architecture
The micro service data is sent to a data buffer queue kafka through log back, a service call chain tracking component zipkin can acquire service call chain related log information from the kafka, log data conversion is carried out, and finally the service call chain related log information is stored in an elastic search component; on the other hand, the log preprocessing log stack component also acquires service log data from the kafka, filters and processes the service log data, and then stores the service log data into the elastic search component; the final log data are stored in the elastic search component, and the log processing module processes and analyzes the log data in the elastic search component and presents a final log analysis result to a user;
mysql database: the mysql database is used for storing service information and interface information base, and contains various information and characteristics of services and interfaces;
a file storage system: the file storage system is used for storing mapping and __ file files of the interface;
and (3) service management: the method is used for managing service information, service addition, service maintenance and micro-service deletion;
interface management: the interface information used for managing the service comprises a new interface, a maintenance interface and a deletion interface;
MOCK service: the interface file processing method comprises the steps of receiving interface request information, calling a URL address mapping system, positioning an interface file, processing interface return information and finally returning MOCK data;
URL mapping: the background service automatically generates an address mapping relation according to the interface information for file positioning;
and (3) file generation: the background service automatically generates mapping and __ file files according to the interface information in the interface management.
Preferably, the presentation layer is used for visually presenting service and interface information and for user interaction.
Preferably, the __ files are files that are uploaded, downloaded or recorded, and are used for receiving and returning interface request file data.
Preferably, the maps are used for storing service interface information data, including mapping information, request and response information of url of the interface.
Compared with the prior art, the invention has the beneficial effects that:
the design of the scheme realizes the functions of service and data simulation, and solves the problem of dependence on external services and data in unit test and integrated test.
Drawings
FIG. 1 is a diagram of the logical relationship of the Mock test platform of the present invention;
FIG. 2 is a system architecture diagram of the present invention;
FIG. 3 is a diagram of a test platform data architecture of the present invention;
FIG. 4 is a flow chart of the Mock platform business of the present invention;
FIG. 5 is a diagram of a log collection architecture of the present invention;
FIG. 6 is a schematic diagram of a service invocation tracking mechanism of the present invention;
FIG. 7 is a system architecture diagram of the microservice application data collection and analysis tool of the present invention;
FIG. 8 is a diagram of a system data architecture of the present invention;
fig. 9 is a technical architecture diagram of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "top," "bottom," "inner," "outer," and the like indicate or are based on the orientation or positional relationship shown in the drawings, merely to facilitate description of the present invention and to simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention.
Examples:
referring to fig. 1-9, the present invention provides a technical solution: the micro-service application test system based on the simulated data generator comprises a presentation layer, an application layer and a data layer, wherein the application layer comprises service management, interface management, URL mapping, MOCK service and file generation, and the data layer comprises a file storage system and a mysql database;
mysql database: the mysql database is used for storing service information and interface information base, and contains various information and characteristics of services and interfaces;
a file storage system: the file storage system is used for storing mapping and __ file files of the interface;
and (3) service management: the method is used for managing service information, service addition, service maintenance, micro-service deletion and the like;
interface management: interface information used for managing service, including newly added interface, maintenance interface, deleted interface, etc.;
MOCK service: the interface file processing method comprises the steps of receiving interface request information, calling a URL address mapping system, positioning an interface file, processing interface return information and finally returning MOCK data;
URL mapping: the background service automatically generates an address mapping relation according to the interface information for file positioning;
and (3) file generation: the background service automatically generates mapping and __ file files according to the interface information in the interface management.
The presentation layer is used for visually presenting service and interface information and for user interaction.
__ files are files that are stored, uploaded, downloaded or recorded, for receiving and returning interface request file data.
The maps are used to store service interface information data, including mapping information, request and response information of url of the interface.
The micro service application test system based on the simulation data generator (Mock) is developed to solve the problem that unit test and integrated test in the micro service application test are dependent on external service and data.
Therefore, the Mock test platform can dynamically construct an analog interface on the Mock test platform through service interface document information configured by a developer, and can return Mock interface data. Other services can access the Mock interface to acquire data with the same format and meaning as the real interface, so that the logic verification is carried out on the own code. In the DevOps workflow of the micro-service application construction, in the unit test and interface test stage, the external API interface of the Mock test platform can be directly called in the test code, and the interface data to be Mock can be returned, so that the purpose of data simulation is achieved. The logical relationship diagram of the Mock test platform is shown in fig. 1.
2) Principle of system implementation
Creating __ files and maps folders on a Mock server, wherein __ files are used for storing uploaded, downloaded or recorded files and used for receiving and returning interface request file data; the maps are used for storing service interface information data, including mapping information, request information, response information and the like of url of the interface.
Typical service interface information file mapping. Json content is as follows:
when the interface/api/testdetails is called in the service unit test and the integration test, the Mock server will find the mapping file corresponding to the interface/api/testdetails according to the mapping relation of the interface route url, and return the content of testdetails.
The Mock test platform also supports returning response content directly in the response structure, such as in a mapping file, writing the response as follows:
when a corresponding Mock interface service request is sent, a "Hello world" will be returned directly.
The Mock test platform is mainly responsible for managing the files of maps and __, managing the service request and returning the request result content file.
3) System architecture design
The micro-service Mock testing platform based on the simulation data generation technology is a micro-service application testing tool based on the simulation data generation technology, which is realized by integrating related key technologies and performing further optimized development on the basis of a micro-service application testing system and a micro-service application key technical scheme. The system is mainly divided into a data layer, an application layer and a presentation layer, and the system architecture is shown in figure 2.
The data layer mainly plays a role in data acquisition and data storage and mainly comprises a file storage system and a MySql-based service and interface information base. The application layer mainly comprises service management, interface management, mock service, URL address mapping and file generation functions. The presentation layer is mainly used for visually presenting service and interface information and for user interaction. Specifically:
mysql database: the mysql database is used for storing service information and interface information base, and contains various information and characteristics of services and interfaces;
a file storage system: the file storage system is used for storing mapping and __ file files of the interface;
and (3) service management: the method is used for managing service information, service addition, service maintenance, micro-service deletion and the like;
interface management: interface information used for managing service, including newly added interface, maintenance interface, deleted interface, etc.;
MOCK service: the interface file processing method comprises the steps of receiving interface request information, calling a URL address mapping system, positioning an interface file, processing interface return information and finally returning MOCK data;
URL mapping: the background service automatically generates an address mapping relation according to the interface information for file positioning;
and (3) file generation: the background service automatically generates mapping and __ file files according to the interface information in the interface management.
The presentation layer is used for visually presenting service and interface information and for user interaction.
__ files are files that are stored, uploaded, downloaded or recorded, for receiving and returning interface request file data.
The maps are used to store service interface information data, including mapping information, request and response information of url of the interface.
4) System data architecture
The test platform data architecture generated based on the Mock technique is shown in fig. 3.
After the service information and the interface basic information are transmitted into the system, the service information and the interface basic information are stored in a database, the system generates mapping.json and __ file files of the corresponding interfaces by means of file generation, and the mapping.json and __ file files are stored in a file storage system. After the user sends out the interface simulation request, the address mapping routing system is called through the dock service to locate the corresponding interface mapping and __ file, the interface return information is processed, and finally dock result data is returned.
5) System business process
The Mock testing platform is divided into a Mock foreground and a Mock background according to design thought and system architecture diagrams, wherein the Mock foreground is mainly responsible for information maintenance of services and maintenance of service interface information, and the Mock background is responsible for constructing a Mock API and an API route mapping function. The specific platform business process is shown in fig. 4;
the method mainly comprises the following steps:
(1) adding service information needing to be simulated by using Mock technology in Mock foreground
(2) And adding the Mock interface information including the name, address, request and response information and the like of the interface into the service.
(3) After the Mock interface information is stored, a Mock API is automatically built in a Mock background, an interface route is automatically added, and the requirements of simulating interface requests and returning interface data are met.
(4) Calling an API interface outside a Mock test platform in a unit test script of a tested server or front end, sending interface request data, and returning result data of the Mock API interface by a Mock background.
(5) And (5) checking the result data, and ending the test.
(2) Developing a microservice application data collection and analysis tool set
In the integrated process of the development, test, operation and maintenance of the DevOps, the monitoring is an important link. And the team feeds back to the planning and development stage according to the monitored data, and influences the research and development process of the whole application. The micro-service application data collection and analysis tool set obtains the log information through the external data output interface and stores and analyzes the log information so as to achieve the purpose of data monitoring.
The micro-service application data collection and analysis tool will be described in terms of five aspects of system design thought, system implementation principle, system architecture design, system data architecture, and system technology architecture.
1) System design concept
The micro-service application data collection and analysis tool is developed to monitor the micro-service in real time, collect service operation log data information, analyze the micro-service system according to the collected data information, and realize continuous optimization for the micro-service application through data feedback.
Micro-services are a distributed architecture, call relationship logic between services is complex, and these characteristics determine that log data analysis is required for micro-service applications from two aspects. On the one hand, the service daily operation log comprises a service operation log, a database operation log, a system fault error log and the like; on the other hand, the service call chain tracks log information, and the service call chain, the time consumption of the chain, the service bottleneck and the like are checked by analyzing the service call chain. Service execution logs and service call chain logs are the content of the micro-service application data collection and analysis tool that is focused on and analyzed.
The micro-service application data collection and analysis tool can collect service running logs and service call chain logs distributed on different servers, and establish a system crash mechanism for preventing the logs from being in a peak period.
2) Principle of system implementation
(1) Distributed architecture log collection principle
In a distributed micro-service application system, logs are stored scattered on different servers. Collecting log data from different services is the underlying work of the data collection tool. In a micro-service architecture, the log back component is typically used by default to record service logs, so an append applicator of the data buffer queue component kafka may be added in a log back configuration to effect collection of log data. The log collection structure is shown in fig. 5;
after the related configuration is finished, related log data is automatically sent to a data buffer area in the running process of the micro-service, and the data can be obtained from the data buffer area and processed by data preprocessing.
(2) Service invocation chain tracking principle
In the micro-service framework, a request initiated by a client is called by a plurality of different service nodes in a back-end system to cooperatively generate a final request result, each front-end request forms a complex distributed service call link, and high delay or error of any ring in the link can cause final failure of the whole request. A distributed service tracking mechanism needs to be introduced to achieve service tracking under a micro-service architecture. The service invocation tracking mechanism principle is shown in fig. 6.
In the micro-service architecture, each service request generates a globally unique TraceID by which the entire call chain can be concatenated, i.e., the system needs to remain in passing the unique TraceID until the request returns while running inside the distributed system.
When a request arrives at each service component, its start, specific procedure and end are marked by span ID. For each Span, it must have a start and end two nodes, and the time delay of the Span is counted by recording the time stamps of the start Span and the end Span.
Each request passes the TraceID and SpanID throughout the call. Each service records the span id attached to the request as a span id (parent span id), and generates its own span id. One Span without PSaniD is the root Span, and can be seen as a call chain entry. So to look at a complete call only needs to find all call records according to the TraceID, and then organize the whole call parent-child relationship through PSpanID and span id.
3) System architecture design
The micro-service application data collection and analysis tool set integrates related technologies and performs further optimized development on the basis of a micro-service application test system and a micro-service application key test technical scheme, so that the micro-service application data collection and analysis tool is realized. The tool is mainly divided into a data layer, an application layer and a presentation layer, and a system architecture diagram is shown in fig. 7.
The data collection layer mainly has the functions of service log data acquisition and data storage and mainly comprises service log data collection, data caching, data preprocessing and data storage. The application layer is mainly used for further processing the preprocessed log data, including statistics, analysis, sequencing, inquiry and the like. The presentation layer is mainly used for visually presenting the service log data analysis results and for user interaction.
And (3) data acquisition: the data acquisition is used for collecting log information such as service running logs, service call logs, database running logs and the like, and log information is obtained by using log back and slot technologies and is sent to the data buffer layer.
Data buffering: the data buffer layer is mainly used for preventing the problem of system breakdown caused by overlarge instantaneous log data volume. As the front buffer, the system can play a role in eliminating the peak of the flow, and solve the problem of mismatching between the peak of the log flow and consumption.
Data preprocessing problem: the number of logs collected by default is huge, and data can be initially processed. And formulating data processing rules, including filtering, converting, deforming and other operations.
And (3) data storage: the log data after preprocessing is stored in a database, including service running logs, service call link logs and the like. The data are stored mainly by utilizing databases such as Hadoop HDFS, mysql and the like.
Log processing: further processing is carried out on the preprocessed log information, including statistics, sequencing, inquiry, analysis and other functions.
System interaction and visualization platform: the method is mainly used for user interaction and data display.
4) System data architecture
A data architecture diagram of the microservice application data acquisition and analysis system is shown in fig. 8.
The data buffer area is used for storing original micro service log data which is not subjected to data preprocessing, the system deletes corresponding log data in the data buffer area after the data preprocessing, and the preprocessed data is stored in the database.
The data storage library comprises log contents such as micro service running logs, database running logs, service call chain tracking logs and the like, and the data can display a final analysis result to a user after passing through the log processing module.
5) System technical architecture
The overall technical architecture of the microservice application data collection and analysis system is shown in fig. 9:
the micro service data is sent to a data buffer queue kafka through log back, and a service call chain tracking component zipkin acquires service call chain related log information from the kafka, performs log data conversion and finally stores the log data into an elastic search component. On the other hand, the log preprocessing logstack component also acquires service log data from the kafka, filters and processes the service log data, and stores the service log data into the elastic search component. The final log data are stored in the elastic search component, and the log processing module processes and analyzes the log data in the elastic search component and presents the final log analysis result to the user.
While the fundamental and principal features of the invention and advantages of the invention have been shown and described, it will be apparent to those skilled in the art that the invention is not limited to the details of the foregoing exemplary embodiments, but may be embodied in other specific forms without departing from the spirit or essential characteristics thereof; the present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (4)
1. The micro service application test system based on the simulation data generator comprises a presentation layer, an application layer and a data layer, and is characterized in that: the application layer comprises service management, interface management, URL mapping, MOCK service and file generation, and the data layer comprises a file storage system and a mysql database;
creating __ files and maps folders on a Mock server, wherein __ files are used for storing uploaded, downloaded or recorded files and used for receiving and returning interface request file data; the maps are used for storing service interface information data, including mapping information, request and response information of url of the interface;
when calling/api/testdetail interfaces in service unit testing and integration testing, the Mock server searches mapping files corresponding to the api/testdetail according to interface routing url mapping relation, and returns the content of testdetail.json, which is a response file to be built in __ files;
the Mock test platform also supports returning response content directly in response structure
The Mock test platform is responsible for managing files of maps and __, managing service requests and returning request result content files;
after service information and interface basic information are transmitted into a system, the service information and the interface basic information are stored in a database, the system generates mapping.json and __ file files of corresponding interfaces in a file generation mode, and the mapping.json and __ file files are stored in a file storage system;
after a user sends an interface simulation request, the address mapping routing system is called through a dock service to locate corresponding interface mapping and __ file files, interface return information is processed, and dock result data is finally returned;
the method comprises the steps that a dock test platform is divided into a dock foreground and a dock background according to design thought and system architecture diagrams, the dock foreground is mainly responsible for information maintenance of services and maintenance of service interface information, and the dock background is responsible for constructing dock API and API route mapping functions;
the method comprises the following steps:
(1) adding service information to be simulated by adopting a Mock technology in a Mock foreground;
(2) adding Mock interface information in the service, wherein the Mock interface information comprises the name, address, request and response information of an interface;
(3) after the Mock interface information is stored, a Mock API is automatically constructed in a Mock background, an interface route is automatically added, and the requirements of simulating interface requests and returning interface data are met;
(4) calling an API interface outside a Mock test platform in a unit test script of a tested server or front end, sending interface request data, and returning result data of the Mock API interface by a Mock background;
(5) and (5) checking the result data, and ending the test.
2. The micro service application test system based on the simulation data generator according to claim 1, wherein: the presentation layer is used for visually presenting service and interface information and for user interaction.
3. The micro service application test system based on the simulation data generator according to claim 1, wherein: the __ files are files which are uploaded, downloaded or recorded and are used for receiving and returning interface request file data.
4. The micro service application test system based on the simulation data generator according to claim 1, wherein: the maps are used for storing service interface information data, including mapping information, request and response information of url of the interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911090197.1A CN110825641B (en) | 2019-11-08 | 2019-11-08 | Micro-service application test system based on simulation data generator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911090197.1A CN110825641B (en) | 2019-11-08 | 2019-11-08 | Micro-service application test system based on simulation data generator |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110825641A CN110825641A (en) | 2020-02-21 |
| CN110825641B true CN110825641B (en) | 2023-12-29 |
Family
ID=69554022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911090197.1A Active CN110825641B (en) | 2019-11-08 | 2019-11-08 | Micro-service application test system based on simulation data generator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110825641B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111752837B (en) * | 2020-06-24 | 2024-01-12 | 北京百度网讯科技有限公司 | View library system test method, device, equipment and storage medium |
| CN112965446B (en) * | 2021-01-29 | 2022-06-14 | 华中科技大学 | Flexible production line control system platform based on micro-service architecture |
| CN113342353B (en) * | 2021-07-01 | 2023-05-23 | 银联商务股份有限公司 | Mock data generation method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105302721A (en) * | 2015-10-29 | 2016-02-03 | 普元信息技术股份有限公司 | System and method for realizing integration testing of centralized analog server in cloud computing platform |
| CN108388519A (en) * | 2018-03-19 | 2018-08-10 | 车智互联（北京）科技有限公司 | Mock test methods based on SpringBoot services |
| CN109902010A (en) * | 2019-02-27 | 2019-06-18 | 苏州浪潮智能科技有限公司 | The method and system that API is tested under a kind of KVM framework |
| CN110147326A (en) * | 2019-05-23 | 2019-08-20 | 四川新网银行股份有限公司 | Micro services automated testing method based on Redis database |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10204035B1 (en) * | 2017-08-07 | 2019-02-12 | Appvance Inc. | Systems, methods and devices for AI-driven automatic test generation |
| US10592402B2 (en) * | 2017-11-20 | 2020-03-17 | International Business Machines Corporation | Automated integration testing with mock microservices |
-
2019
- 2019-11-08 CN CN201911090197.1A patent/CN110825641B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105302721A (en) * | 2015-10-29 | 2016-02-03 | 普元信息技术股份有限公司 | System and method for realizing integration testing of centralized analog server in cloud computing platform |
| CN108388519A (en) * | 2018-03-19 | 2018-08-10 | 车智互联（北京）科技有限公司 | Mock test methods based on SpringBoot services |
| CN109902010A (en) * | 2019-02-27 | 2019-06-18 | 苏州浪潮智能科技有限公司 | The method and system that API is tested under a kind of KVM framework |
| CN110147326A (en) * | 2019-05-23 | 2019-08-20 | 四川新网银行股份有限公司 | Micro services automated testing method based on Redis database |
Non-Patent Citations (3)
| Title |
|---|
| 孟燕 ; .自动化测试技术中Mock框架的建设与应用.武汉理工大学学报(信息与管理工程版).2017,(第03期),全文. * |
| 李明 ; 王伟 ; 张栋栋 ; .传统架构升级微服务的设计与实现.科技传播.2019,(第10期),全文. * |
| 潘诗瑶 ; 黄建明 ; .Web应用系统中的MOCK测试技术.软件.2016,(第12期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110825641A (en) | 2020-02-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309071B (en) | Test code generation method and module, and test method and system | |
| Engel et al. | Evaluation of microservice architectures: A metric and tool-based approach | |
| CN110825641B (en) | Micro-service application test system based on simulation data generator | |
| CN110321113B (en) | Integrated assembly line system taking project batches as standards and working method thereof | |
| US8930918B2 (en) | System and method for SQL performance assurance services | |
| KR101785959B1 (en) | Columnar storage representations of records | |
| CN106021079A (en) | A Web application performance testing method based on a user frequent access sequence model | |
| CN112347071B (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN108255712A (en) | The test system and test method of data system | |
| CN107301205A (en) | A kind of distributed Query method in real time of big data and system | |
| CN107103064B (en) | Data statistical method and device | |
| Cheah et al. | Milieu: Lightweight and configurable big data provenance for science | |
| WO2023202497A1 (en) | Method for full-link tracking of transaction, and native distributed database | |
| CN111753034A (en) | One-stop type geographical big data platform | |
| US11321318B2 (en) | Dynamic access paths | |
| CN107025167A (en) | The method and apparatus for carrying out data-flow analysis using compiler type information in processor trace log | |
| CN112700083A (en) | Method and device for constructing scene of resource comprehensive utilization and service index system | |
| Volk et al. | Ask the Right Questions: Requirements Engineering for the Execution of Big Data Projects. | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| Sfaxi et al. | Babel: a generic benchmarking platform for Big Data architectures | |
| CN111125045B (en) | Lightweight ETL processing platform | |
| Mishra et al. | Challenges in big data application: a review | |
| Hryszko et al. | Bottlenecks in software defect prediction implementation in industrial projects | |
| CN103186551B (en) | Exception analysis method and analogue system based on web application platform | |
| Baquero et al. | A Framework to Support Business Process Analytics. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |