CN110750458A - Big data platform testing method and device, readable storage medium and electronic equipment - Google Patents
Big data platform testing method and device, readable storage medium and electronic equipment Download PDFInfo
- Publication number
- CN110750458A CN110750458A CN201911006343.8A CN201911006343A CN110750458A CN 110750458 A CN110750458 A CN 110750458A CN 201911006343 A CN201911006343 A CN 201911006343A CN 110750458 A CN110750458 A CN 110750458A
- Authority
- CN
- China
- Prior art keywords
- big data
- test
- data platform
- client node
- jmeter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 211
- 238000000034 method Methods 0.000 claims abstract description 38
- 238000004891 communication Methods 0.000 claims abstract description 14
- 238000011056 performance test Methods 0.000 claims abstract description 10
- 238000009434 installation Methods 0.000 claims description 33
- 238000012544 monitoring process Methods 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 230000000977 initiatory effect Effects 0.000 claims 1
- 230000008569 process Effects 0.000 description 15
- 239000000306 component Substances 0.000 description 14
- 238000013515 script Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000013507 mapping Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000007418 data mining Methods 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000010998 test method Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Prevention of errors by analysis, debugging or testing of software
- G06F11/3668—Testing of software
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Prevention of errors by analysis, debugging or testing of software
- G06F11/3668—Testing of software
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a big data platform testing method, a big data platform testing device, a readable storage medium and electronic equipment, and the performance testing of the big data platform is realized through a JMeter. The method comprises the following steps: a client node receives a test starting instruction input by a user to start a performance test tool JMeter installed on the client node; the client node acquires a pre-stored connection class and establishes communication with a target big data platform server; the client node analyzes the test case pre-stored on the client node through the JMeter to obtain the parameters for running the test case; and the client node starts a test thread according to the obtained parameters of the running test case to send a data read-write request to the target big data platform server, and the target big data platform server performs testing according to the data read-write request. The scheme can improve the working efficiency of testing personnel when testing different large data platforms.
Description
Technical Field
The application relates to the technical field of big data, in particular to a big data platform testing method and device, a readable storage medium and electronic equipment.
Background
The big data technology can mine hidden information and knowledge in large-scale data, so that basis is provided for human social and economic activities. The implementation of big data technology usually depends on a big data platform, and for the big data platform, the big data platform has functions of data acquisition, data processing, data storage, data mining and the like. Therefore, the performance of the big data platform is very important in data processing of the big data platform, and the performance condition of the big data platform can be obtained by performing pressure test on the big data platform.
In the prior art, when pressure testing of big data is performed, various testing tools carried on a big data platform are used for performing pressure testing on the big data platform. In the test method, the test tools need to be integrated by programming, and because the test tools carried by different big data platforms are different, test scripts need to be written specifically for different big data platforms to integrate the test tools carried by the big data platforms, so the test method has very low universality, and has large workload and very low test efficiency when different big data platforms need to be tested.
Disclosure of Invention
In order to overcome at least the above-mentioned deficiencies in the prior art, an object of the present application is to provide a big data platform testing method applied to a client node, the method comprising:
receiving a test starting instruction input by a user to start a performance test tool JMeter installed on the client node;
acquiring a pre-stored connection class for connecting a target big data platform server from the client node, and establishing communication with the target big data platform server through the connection class;
analyzing the test case pre-stored on the client node through the JMeter to obtain parameters for operating the test case;
and starting a test thread according to the obtained parameters for running the test case to send a data read-write request to the target big data platform server so that the target big data platform server performs a test based on the data read-write request.
Optionally, before receiving a test start instruction input by a user to start a performance testing tool JMeter installed on the client node, the method further includes:
acquiring a JMeter installation package;
decompressing the JMeter installation package to obtain a JMeter installation file;
and acquiring a first installation instruction input by a user, and installing JMeter on the client node by using the decompressed JMeter installation file.
Optionally, the method further comprises:
acquiring the number of threads input by a user, time parameters including the cycle times and/or the running duration of the threads, the request times of each thread for sending a request, the test type of the performance type of a test server and check data for identifying whether to check a test result;
and generating and storing a test case according to the thread number, the time parameter, the request times, the test type and the viewing data.
Another object of an embodiment of the present application is to provide a big data platform testing method, which is applied to a testing system including a client node and a target big data testing platform, and the method includes:
the client node receives a test starting instruction input by a user to start a performance test tool JMeter installed on the client node;
the client node acquires a pre-stored connection class for connecting a target big data platform server from the client node, so as to establish communication with the target big data platform server through the connection class;
the client node analyzes the test case pre-stored on the client node through the JMeter to obtain the parameters for operating the test case;
the client node starts a test thread according to the obtained parameters for running the test case and sends a data read-write request to the target big data platform server;
and the target big data service platform receives the data read-write request sent by the client node and tests according to the data read-write request.
Optionally, the method further comprises:
acquiring test data consisting of various types of data to be stored;
inputting the test data into a Map/Reduce frame which is pre-deployed on the target big data platform service end so as to classify and store the test data to each service node on the target big data platform service end;
and obtaining a verification result of whether the classified test data is stored on each service node.
Optionally, the big data platform on the target big data platform server is a Ceph big data platform, and before receiving a test start instruction input by a user to start a performance test tool JMeter installed on the client node, the method further includes:
the target big data platform server side obtains a download address of the Ceph software input by a user;
the target big data platform server side acquires the Ceph software according to the download address of the Ceph software;
the target big data platform server side installs the Ceph software according to a second installation instruction input by a user;
the target big data platform server side obtains a configuration file for configuring a Ceph environment, and configures the Ceph environment according to the configuration file, wherein the Ceph environment comprises: a monitoring node and a data storage node.
It is another object of the embodiments of the present application to provide a big data platform testing apparatus, which is applied to a client node, and the apparatus includes:
the receiving module is used for receiving a test starting instruction input by a user so as to start a performance test tool JMeter installed on the client node;
the acquisition module is used for acquiring a pre-stored connection class for connecting a target big data platform server from the client node so as to establish communication with the target big data platform server through the connection class;
the analysis module is used for analyzing the test cases stored in the client node in advance through the JMeter to obtain parameters for operating the test cases;
and the test module is used for starting a test thread according to the obtained parameters for running the test case to send a data read-write request to the target big data platform server so that the target big data platform server responds to the data read-write request to perform testing.
Optionally, the apparatus further comprises: a software installation module specifically configured to:
acquiring a JMeter installation package;
decompressing the JMeter installation package to obtain a JMeter installation file;
and acquiring a first installation instruction input by a user, and installing JMeter on the client node by using the decompressed JMeter installation file.
It is also an object of the embodiments of the present application to provide a readable storage medium, which stores an executable program, and when executing the executable program, a processor implements the method according to any one of the present applications.
It is another object of the embodiments of the present application to provide an electronic device, which includes a memory and a processor, the memory is connected to the processor, the memory stores an executable program, and the processor implements the method according to any of the present application when executing the executable program.
Compared with the prior art, the method has the following beneficial effects:
according to the big data platform testing method, the big data platform testing device, the readable storage medium and the electronic equipment, the performance testing tool JMeter is installed on the client node, the JMeter is operated to obtain the performance testing tool JMeter to establish communication between the client and the target big data platform, the parameters for operating the test case are obtained from the test case, and therefore the data read-write request is sent to the big data platform server side, and the big data platform server side can conduct testing based on the data read-write request. The connection adopts JMeter to test the big data platform, the characteristics that a JMeter tool supports secondary development are utilized to obtain and analyze a test case, connection is established with the big data platform server, and then a data read-write request is sent according to analyzed test case data, so that an interface function of the JMeter is called in the running test case to call an interface of the server to be tested to realize testing. In the whole process, the JMeter platform can run in a multi-language environment, so that when different big data platforms are tested, only the corresponding interface of the big data platform needs to be called through the JMeter, therefore, the scheme of the embodiment has stronger universality, and when different big data platforms are tested, the workload can be reduced, and the efficiency can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
FIG. 1 is a block diagram illustrating a first schematic structure of a test system provided in an embodiment of the present application;
FIG. 2 is a block diagram schematically illustrating a structure of an electronic device according to an embodiment of the present disclosure;
FIG. 3 is a schematic flow chart of a big data platform testing method provided in an embodiment of the present application;
FIG. 4 is a block diagram of a second schematic structure of a test system provided in the embodiments of the present application;
fig. 5 is a block diagram schematically illustrating a structure of a big data platform testing apparatus according to an embodiment of the present application.
Icon: 100-target big data platform server; 200-a client terminal; 300-a network; 400-an electronic device; 410-big data platform testing device; 420-a memory; 430-a processor; 411-a receiving module; 412-an acquisition module; 413-an analysis module; 414-test module; 415-software installation module.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
In the description of the present application, it is further noted that, unless expressly stated or limited otherwise, the terms "disposed," "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present application can be understood in a specific case by those of ordinary skill in the art.
With the development of social economy, the data volume is increased dramatically, how to extract useful information from numerous data is very significant, and large data technologies are produced, and a large number of new technologies about large data acquisition, storage, processing analysis or visualization and the like are developed almost every year. For example, the Hadoop service adopting the Hadoop big data technology provides big data tools for various links such as data acquisition, data processing, data storage, data mining and the like, and comprises a complete data storage and data processing scheme, so that the Hadoop service becomes an industrial standard of big data. The Hadoop big data platform for arranging the Hadoop service is based on an open source community, has a good ecosystem, comprises a plurality of complex distributed modules, is interdependent among the modules, and has platform and large-scale characteristics, for example, the Hadoop big data platform mainly comprises core components such as HDFS, MapReduce, Hbase, Zookeeper, Hive and the like, and further comprises frameworks such as Spark, Flume and the like for being fused with other services, so that great challenges are brought to the test of the big data platform.
The Hadoop service is provided with several testing components, and the component sequences can test the carrying platform of the Hadoop big data service from multiple angles, such as a cluster performance testing component TestDFSIO, a basic function testing component MRBench and a load testing component NNBench. TestDFSIO is used to test IO (Input Output) performance of HDFS (Hadoop Distributed File System). NNBench is used to test the load of the NameNode, generate many HDFS related requests, and apply a large pressure to the single server node NameNode. The MRBench repeatedly executes a small job a plurality of times for checking whether the operation of the small job on the cluster is repeated and the efficiency of the operation.
Other big data services, such as Redis service has a Redis-benchmark component for testing its performance, Ceph service has a Rados Bench component for testing performance, and so on.
Because the test components carried by different big data services can only carry out single type of tests generally, to realize the test of multiple performances, the test components on the big data services are often integrated by writing programs, so that all the components work according to certain logic. Different big data platforms have to be purposefully written for performance testing, since the testing tools are different from one big data platform to another. This will certainly take a lot of time to program when testing different big data services, resulting in a heavy workload and inefficient testing for the tester.
In addition, the big data service self-contained test component also needs a tester to write a specific test script to collect various data result data so as to verify the correctness of the test result, and write a visual script to show the test result of the big data.
In order to solve the above problem, an embodiment of the present application provides a testing system, please refer to fig. 1, where the testing system includes a target big data platform server 100 and at least one client terminal 200. The target big data platform server 100 and the client terminal 200 are respectively connected to a network 300. Wherein, the client node is configured on a client terminal 200, the client terminal 200 can be an electronic device 400, and when there are a plurality of client nodes, a distributed structure is formed among the client nodes. The target big data platform service end 100 may also be composed of a plurality of service terminals, and each service terminal corresponds to one service node.
Referring to fig. 2, fig. 2 is a schematic block diagram of a structure of an electronic device 400 according to an embodiment of the present disclosure, where the electronic device 400 includes a big data platform testing apparatus 410, a memory 420 and a processor 430, and the memory 420 and the processor 430 are electrically connected to each other directly or indirectly for implementing data interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The big data platform testing apparatus 410 includes at least one software function module which can be stored in the memory 420 in the form of software or firmware (firmware) or solidified in an Operating System (OS) of the electronic device 400. The processor 430 is used for executing executable modules stored in the memory 420, such as software functional modules and computer programs included in the big data platform testing device 410.
The embodiment of the present application further provides a method applied to the above test system, which is explained in detail below with reference to specific steps, please refer to fig. 3, and the method includes steps S110 to S150.
Step S110, the client node receives a test starting instruction input by a user and starts the JMeter.
Specifically, the client node receives a test starting instruction input by a user to start a performance testing tool JMeter installed on the client node.
Step S120, the client node establishes communication with the target big data platform server 100.
Specifically, the client node acquires a pre-stored connection class for connecting the target big data platform server 100 from the client node, so as to establish communication with the target big data platform server 100 through the connection class.
Step S130, the client node analyzes the test case pre-stored on the client node through the JMeter to obtain the parameters for running the test case.
In step S140, the client node sends a data read/write request to the target big data platform server 100.
Specifically, the client node starts a test thread according to the obtained parameters for running the test case and sends a data read-write request to the target big data platform server 100.
And step S150, the target big data platform server 100 tests according to the data read-write request.
Specifically, the target big data platform server 100 receives the data read-write request sent by the client node, and performs a test according to the data read-write request.
The embodiment is used for testing the target big data platform service end 100 through the JMeter, and the JMeter can be used for various testing functions and supports secondary development, so that the JMeter can be applied to big data platform testing. Since the JMeter itself can be compatible with systems in multiple language environments, when the JMeter is used for testing, only the part of the test script corresponding to the interface provided by the tested end (for example, the target big data platform server 100 in the present application) needs to be changed, so that the JMeter can be conveniently used among multiple big data platforms, the workload when different big data platforms are tested can be reduced, and the working efficiency can be improved.
The JMeter basic principle is that a thread pool is established, multithreading operation is delayed to generate a large amount of load, correctness of results is verified through assertion in the operation process, and test results are recorded through a monitor. JMeter includes the most basic three components, respectively: thread groups (scene setting), samplers (logic control), listeners (data reporting).
And (3) thread group: all tasks are based on the thread group, and the number of opened threads represents the number of concurrent users; JMeter is implemented in Java and uses a Java thread to simulate a user, so a thread group (ThreadGroup) means a group of users, in other words, a thread group is a group of virtual users (virtual users) that simulate accessing a system under test (e.g., the target big data platform server 100).
The sampler includes: a time parameter Ramp-Up Period comprising the number of cycles and/or run length of a thread, a test class Sampler, and an assertion. Ramp-Up Period: all tests are completed within such a large amount of time, for example, 2 threads are opened, and if the Ramp-Up Period is 3, the login of 2 users is completed within 3 seconds; the interval for each thread is 1.5 seconds. Sampler: all test tasks are samplers, i.e. the category of any test task is Sampler, such as HTTP request, JDBC request, FTP request; assertion: the Sampler test is performed to determine if it is correct, and belongs to one of the post-processors 430. A listener: and counting and displaying the request result of Sampler.
In this embodiment, before step S110, data for responding to the client node stored in the target big data platform server 100 may also be collected, for example, js (javascript) code may be embedded in the web page, and the data is collected and transmitted to nginnx (Engine x, HTTP and reverse proxy web server), and the nginnx processes and stores the result. The test data may be taken from web portals, video media, search engines, mainstream social media, such as but not limited to exposure of advertisements, click information, and the like.
Optionally, in this embodiment, before step S110, the method further includes installing a JMeter, specifically, first obtaining a JMeter installation package; decompressing the JMeter installation package to obtain a JMeter installation file; and finally, acquiring a first installation instruction input by a user, and installing JMeter on the client node by using the decompressed JMeter installation file.
Referring to fig. 4, when there are multiple client nodes installed with JMeter, each client node elects a master node master and at least one slave node slave, obtains data required for installing JMeter from outside through the master node and distributes the data to each slave node slave, and each slave node slave then sends a Request to a target big data platform server 100 installed with big data service.
Optionally, in this embodiment, before step S110, a big data platform may also be configured first.
For example, the big data platform on the target big data platform server 100 may be a Ceph big data platform. The specific steps for configuring the big data platform are as follows: firstly, the target big data platform server 100 obtains the download address of the Ceph software input by the user, namely, yum source is configured, and the Ceph source is added. The target big data platform server 100 acquires the Ceph software according to the download address of the Ceph software; then, the target big data platform server 100 installs the Ceph software according to a second installation instruction input by the user (each service terminal installs the Ceph software); finally, the target big data platform server 100 obtains a configuration file for configuring a Ceph environment, and configures the Ceph environment according to the configuration file, where the Ceph environment includes: a monitoring node and a data storage node.
The concrete steps of configuring the Ceph environment according to the configuration file comprise: firstly, deploying monitor nodes; (the first step of deploying Ceph is to run the monitors of the cluster, the Ceph cluster needs at least one monitor to run normally, but generally, three or more monitors are set for high availability, for example, the monitors map which three monitors jointly form the cluster is installed, the monitors can control multiple behaviors of the cluster, such as the number of copies of each pool, the number of processes on each OSD, the inter-OSD heartbeat interval, whether authentication (Cephx) is enabled, and the like, and these configuration items have default values, and we can adjust optimization as required). And then, sequentially creating a Ceph. conf file for storing configuration information of the monitoring node, creating a monitor map, initializing a monitor directory, acquiring the monitor map and key, updating the Ceph. conf file, and starting a Ceph-mon process for maintaining a primary copy mapping of the client node.
After the monitors run, the cluster already has a default CRUSH Map, since no OSD has been added, and therefore there is no mapping in this CRUSH Map. The cluster state can be checked by using Ceph-s, and at this time, the current cluster is in an Error reporting state, and 0 OSD exists, so that an OSD node needs to be installed.
The specific process of installing the OSD nodes is as follows: firstly, adding OSD nodes; and then, sequentially carrying out the steps of initializing an OSD directory, changing a journal (log) partition, registering OSD.0, adding CRUSH Map and starting a Ceph-OSD process.
After the OSD nodes are installed, Ceph is deployed, and at this time, the test data may be stored in the target big data platform server 100. Then the consistency, validity and accuracy of the data are checked to determine the quality of the data quality of the test data, namely whether the stored data is tested correctly.
Optionally, in this embodiment, after the target big data platform service end 100 is configured, service logic verification may be performed on each service terminal of the target big data platform service end 100, that is, whether each service terminal can be used normally, for example, whether data can be stored correctly.
In the specific verification, it may be verified whether the service logic on each node (the Ceph service respectively deployed on each machine) is correct first, and then verified whether the service logic of the plurality of nodes is correct. For example, "Map/Reduce" may be used for authentication. Map/Reduce is a Hadoop computing framework, and the basic principle is as follows: a Map/Reduce Job (Job) will typically split the input dataset into several independent data blocks, which are processed in a completely parallel manner by the Map Task (Task). Map/Reduce sorts the output of Map and inputs the result to Reduce task. Typically both the input and output of a job will be stored in a file system. The whole framework is responsible for scheduling and monitoring tasks and re-executing tasks that have failed. Therefore, the specific process of performing business logic verification by Map/Reduce is as follows: firstly, acquiring test data composed of various types of data to be stored, for example, data for responding to a client node, which is stored in a target big data platform server 100; then, the test data is input into a programming model Map/Reduce pre-deployed on the target big data platform service end 100, so that the test data is classified and stored in each service node on the target big data platform service end 100, and finally, a verification result of whether the classified test data is stored in each service node is obtained.
Map/Reduce checks whether the mapping process (the mapping process is to classify test data packets, and the mapping process supports distributed traversal of each server and classifies the test data packets) is realized, checks whether the aggregation and separation rules of the data are correct after the mapping process (namely, the simplification process is to calculate in packets, and the simplification process also supports distributed data and calculates directly in the classification process).
After the client node and the target big data platform server 100 are deployed, the communication interface of the target big data platform server 100 to be tested in the test script can be determined according to the target big data platform server 100 to be tested.
Firstly, a script of a connection class can be written for a client node to establish communication with a target big data platform service end 100, the script including the connection class is packaged into a jar packet and stored in a/lib/ext/directory decompressed by a JMeter, so that the JMeter can find the jar packet to be tested according to a class name class set by a test case. After that, the client node continues to run the test script, and in the test script, through the JavaSamplerClient interface provided by JMeter, environment initialization is carried out, and the read and write interfaces provided by the Ceph API are called.
Optionally, before step S110, after the JMeter is installed, a test case may be prepared and saved on the client node. The test cases for storing on the client node may be obtained from outside by a user, or may be obtained by, for example, a user through a graphical user interface of JMeter, the number of threads input by the user, a time parameter including the number of cycles and/or the running time of the thread, the number of requests sent by each thread, a test category of a performance category of the test server, and viewing data identifying whether to view a test result. The JMeter acquires the Number of Threads (equivalent to a user interacting with a service end) input by a user, a time parameter Ramp-Up Period (in seconds) comprising the cycle times and/or the running time of the Threads, the request times Loop Count of each thread sending request, a test type Sampler of the performance type of a test server and check data Report for identifying whether to check the test result; and generating and storing a test case according to the thread number, the time parameter, the request times, the test type and the viewing data.
According to the scheme, when a plurality of client nodes exist, large concurrency test can be realized. For example, in a read-write test of a Hadoop big data platform, a pressure test tool is JMeter, the JMeter is deployed on 4 machines, a big data Ceph server (a tested object) is deployed on 10 machines, and if the concurrency number of each machine is set to 1000, the JMeter starts a test and simultaneously performs a pressure test on the read or write operation of the target big data platform server 100 by 4000 concurrency numbers.
In this embodiment, the client node may also be detected in real time, for example, a real-time monitor may be obtained inside the console by using a monitoring test tool Taurus. Taurus can act as a JMX script runner and provide many other functions, including real-time monitoring. System throughput, response time and system performance, CPU, memory, disk, network 300 throughput, etc. may be viewed in real time. In this embodiment, a test may also be run on the BlazeMeter, monitoring the above-mentioned various indicators of the client node through an advanced color report.
After the scheme is adopted for testing, the running state of the large data platform server can be obtained, the test report is obtained at the client node, and developers can find out resources and system bottlenecks, optimize configuration and adjust resource allocation by analyzing the system performance through the test report.
The scheme can also be applied to load tests of other big data tools, such as Hive, Hbase, Spark SQL Requests, Redis, Alluxio and the like.
Another objective of the present embodiment is to provide a big data platform testing apparatus 410, please refer to fig. 5, which includes a receiving module 411, an obtaining module 412, a parsing module 413, and a testing module 414. The big data platform testing apparatus 410 includes a software function module that can be stored in the memory 420 in the form of software or firmware or solidified in an Operating System (OS) of the electronic device 400.
The receiving module 411 is configured to receive a test starting instruction input by a user to start a performance testing tool JMeter installed on the client node.
The receiving module 411 in this embodiment is used to execute step S110, and for the specific description of the receiving module 411, reference may be made to the description of step S110.
An obtaining module 412, configured to obtain a pre-stored connection class for connecting to the target big data platform server 100 from the client node, so as to establish communication with the target big data platform server 100 through the connection class.
The obtaining module 412 in this embodiment is configured to perform the step S120, and for a detailed description of the obtaining module 412, reference may be made to the description of the step S120.
And the analysis module 413 is configured to analyze the test case pre-stored in the client node through the JMeter to obtain a parameter for running the test case.
The parsing module 413 in this embodiment is configured to perform the step S130, and the detailed description about the parsing module 413 may refer to the description about the step S130.
The testing module 414 is configured to start a testing thread according to the obtained parameter for running the test case to send a data read-write request to the target big data platform server 100, so that the target big data platform server 100 responds to the data read-write request to perform a test.
The testing module 414 in this embodiment is used to execute step S140, and the detailed description about the testing module 414 may refer to the description about the step S140.
Optionally, the apparatus further comprises: a software installation module 415, wherein the software installation module 415 is specifically configured to:
acquiring a JMeter installation package, and then decompressing the JMeter installation package to acquire a JMeter installation file; and acquiring a first installation instruction input by a user, and installing JMeter on the client node by using the decompressed JMeter installation file.
It is another object of the embodiments of the present application to provide a readable storage medium, where an executable program is stored, and when the executable program is executed, the processor 430 implements the method according to any one of the embodiments.
According to the scheme, automatic big data performance test can be achieved, and the performance of a big data system architecture can be effectively evaluated. The method and the device promote the development of big data technology, improve the testing efficiency of testers and reduce the complexity of big data testing. Meanwhile, the system deployment strategy is more effectively adjusted, other strategies are carried, the system architecture and the performance are improved, the performance bottleneck of the big data system architecture can be determined, and therefore system configuration and resource allocation are optimized. The above description is only for various embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and all such changes or substitutions are included in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A big data platform testing method is applied to a client node, and comprises the following steps:
receiving a test starting instruction input by a user to start a performance test tool JMeter installed on the client node;
acquiring a pre-stored connection class for connecting a target big data platform server from the client node, and establishing communication with the target big data platform server through the connection class;
analyzing the test case pre-stored on the client node through the JMeter to obtain parameters for operating the test case;
and starting a test thread according to the obtained parameters for running the test case to send a data read-write request to the target big data platform server so that the target big data platform server performs a test based on the data read-write request.
2. The method of claim 1, wherein prior to receiving a user-entered test initiation instruction to initiate a performance testing tool JMeter installed on the client node, the method further comprises:
acquiring a JMeter installation package;
decompressing the JMeter installation package to obtain a JMeter installation file;
and acquiring a first installation instruction input by a user, and installing JMeter on the client node by using the decompressed JMeter installation file.
3. The method of claim 1, further comprising:
acquiring the number of threads input by a user, time parameters including the cycle times and/or the running duration of the threads, the request times of each thread for sending a request, the test type of the performance type of a test server and check data for identifying whether to check a test result;
and generating and storing a test case according to the thread number, the time parameter, the request times, the test type and the viewing data.
4. A big data platform testing method is applied to a testing system comprising a client node and a target big data testing platform, and comprises the following steps:
the client node receives a test starting instruction input by a user to start a performance test tool JMeter installed on the client node;
the client node acquires a pre-stored connection class for connecting a target big data platform server from the client node, so as to establish communication with the target big data platform server through the connection class;
the client node analyzes the test case pre-stored on the client node through the JMeter to obtain the parameters for operating the test case;
the client node starts a test thread according to the obtained parameters for running the test case and sends a data read-write request to the target big data platform server;
and the target big data service platform receives the data read-write request sent by the client node and tests according to the data read-write request.
5. The method of claim 4, further comprising:
acquiring test data consisting of various types of data to be stored;
inputting the test data into a Map/Reduce frame which is pre-deployed on the target big data platform service end so as to classify and store the test data to each service node on the target big data platform service end;
and obtaining a verification result of whether the classified test data is stored on each service node.
6. The method as claimed in claim 4, wherein the big data platform on the target big data platform server is a Ceph big data platform, and before receiving a test start instruction input by a user to start a performance test tool JMeter installed on the client node, the method further comprises:
the target big data platform server side obtains a download address of the Ceph software input by a user;
the target big data platform server side acquires the Ceph software according to the download address of the Ceph software;
the target big data platform server side installs the Ceph software according to a second installation instruction input by a user;
the target big data platform server side obtains a configuration file for configuring a Ceph environment, and configures the Ceph environment according to the configuration file, wherein the Ceph environment comprises: a monitoring node and a data storage node.
7. A big data platform testing apparatus, applied to a client node, the apparatus comprising:
the receiving module is used for receiving a test starting instruction input by a user so as to start a performance test tool JMeter installed on the client node;
the acquisition module is used for acquiring a pre-stored connection class for connecting a target big data platform server from the client node so as to establish communication with the target big data platform server through the connection class;
the analysis module is used for analyzing the test cases stored in the client node in advance through the JMeter to obtain parameters for operating the test cases;
and the test module is used for starting a test thread according to the obtained parameters for running the test case to send a data read-write request to the target big data platform server so that the target big data platform server responds to the data read-write request to perform testing.
8. The apparatus of claim 7, further comprising: a software installation module specifically configured to:
acquiring a JMeter installation package;
decompressing the JMeter installation package to obtain a JMeter installation file;
and acquiring a first installation instruction input by a user, and installing JMeter on the client node by using the decompressed JMeter installation file.
9. A readable storage medium, characterized in that the readable storage medium stores an executable program, which when executed by a processor implements the method according to any one of claims 1-3.
10. An electronic device comprising a memory and a processor, the memory being coupled to the processor, the memory having stored therein an executable program, the processor, when executing the executable program, implementing the method of any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911006343.8A CN110750458A (en) | 2019-10-22 | 2019-10-22 | Big data platform testing method and device, readable storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911006343.8A CN110750458A (en) | 2019-10-22 | 2019-10-22 | Big data platform testing method and device, readable storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110750458A true CN110750458A (en) | 2020-02-04 |
Family
ID=69279326
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911006343.8A Pending CN110750458A (en) | 2019-10-22 | 2019-10-22 | Big data platform testing method and device, readable storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750458A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111382080A (en) * | 2020-03-16 | 2020-07-07 | 深圳市吉祥腾达科技有限公司 | Stability test method for equipment cloud management platform system |
| CN111708738A (en) * | 2020-05-29 | 2020-09-25 | 深圳市瑞驰信息技术有限公司 | Method and system for realizing data inter-access between hdfs of hadoop file system and s3 of object storage |
| CN111881060A (en) * | 2020-09-28 | 2020-11-03 | 北京智源人工智能研究院 | Interface performance testing method and device and electronic equipment |
| CN112134754A (en) * | 2020-09-18 | 2020-12-25 | 中移（杭州）信息技术有限公司 | Pressure testing method and device, network equipment and storage medium |
| CN112418930A (en) * | 2020-11-23 | 2021-02-26 | 恩亿科(北京)数据科技有限公司 | Test method, test system and computer equipment |
| CN112527686A (en) * | 2020-12-29 | 2021-03-19 | 平安普惠企业管理有限公司 | Interface performance test method, device, equipment and storage medium |
| CN112749089A (en) * | 2021-01-13 | 2021-05-04 | 上海幻电信息科技有限公司 | Test method and system for probability related function module |
| CN112835334A (en) * | 2020-12-31 | 2021-05-25 | 广州明珞装备股份有限公司 | Industrial data platform testing method and device, computer equipment and storage medium |
| CN112988424A (en) * | 2021-03-22 | 2021-06-18 | 广州虎牙科技有限公司 | Independent process access implementation method and device, electronic equipment and readable storage medium |
| CN113010392A (en) * | 2021-02-08 | 2021-06-22 | 建信金融科技有限责任公司 | Testing method, device, equipment, storage medium and system for big data platform |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045710A (en) * | 2015-06-30 | 2015-11-11 | 吉林大学 | Method for automatically generating test data in cloud computing environment |
| CN106470133A (en) * | 2015-08-20 | 2017-03-01 | 阿里巴巴集团控股有限公司 | System pressure method of testing and device |
| CN107608901A (en) * | 2017-10-20 | 2018-01-19 | 北京京东金融科技控股有限公司 | Method of testing and device based on Jmteter, storage medium, electronic equipment |
| CN109344076A (en) * | 2018-10-22 | 2019-02-15 | 浙江执御信息技术有限公司 | A kind of automatic interface testing method and its device based on JMeter and Jenkins |
| CN109597761A (en) * | 2018-12-03 | 2019-04-09 | 四川长虹电器股份有限公司 | A kind of automatic interface testing method based on Jmeter |
-
2019
- 2019-10-22 CN CN201911006343.8A patent/CN110750458A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045710A (en) * | 2015-06-30 | 2015-11-11 | 吉林大学 | Method for automatically generating test data in cloud computing environment |
| CN106470133A (en) * | 2015-08-20 | 2017-03-01 | 阿里巴巴集团控股有限公司 | System pressure method of testing and device |
| CN107608901A (en) * | 2017-10-20 | 2018-01-19 | 北京京东金融科技控股有限公司 | Method of testing and device based on Jmteter, storage medium, electronic equipment |
| CN109344076A (en) * | 2018-10-22 | 2019-02-15 | 浙江执御信息技术有限公司 | A kind of automatic interface testing method and its device based on JMeter and Jenkins |
| CN109597761A (en) * | 2018-12-03 | 2019-04-09 | 四川长虹电器股份有限公司 | A kind of automatic interface testing method based on Jmeter |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111382080A (en) * | 2020-03-16 | 2020-07-07 | 深圳市吉祥腾达科技有限公司 | Stability test method for equipment cloud management platform system |
| CN111708738A (en) * | 2020-05-29 | 2020-09-25 | 深圳市瑞驰信息技术有限公司 | Method and system for realizing data inter-access between hdfs of hadoop file system and s3 of object storage |
| CN111708738B (en) * | 2020-05-29 | 2023-11-03 | 深圳市瑞驰信息技术有限公司 | Method and system for realizing interaction of hadoop file system hdfs and object storage s3 data |
| CN112134754A (en) * | 2020-09-18 | 2020-12-25 | 中移（杭州）信息技术有限公司 | Pressure testing method and device, network equipment and storage medium |
| CN111881060A (en) * | 2020-09-28 | 2020-11-03 | 北京智源人工智能研究院 | Interface performance testing method and device and electronic equipment |
| CN112418930A (en) * | 2020-11-23 | 2021-02-26 | 恩亿科(北京)数据科技有限公司 | Test method, test system and computer equipment |
| CN112418930B (en) * | 2020-11-23 | 2024-02-27 | 恩亿科(北京)数据科技有限公司 | Test method, system and computer equipment |
| CN112527686A (en) * | 2020-12-29 | 2021-03-19 | 平安普惠企业管理有限公司 | Interface performance test method, device, equipment and storage medium |
| CN112835334B (en) * | 2020-12-31 | 2022-05-27 | 广州明珞装备股份有限公司 | Industrial data platform testing method and device, computer equipment and storage medium |
| CN112835334A (en) * | 2020-12-31 | 2021-05-25 | 广州明珞装备股份有限公司 | Industrial data platform testing method and device, computer equipment and storage medium |
| CN112749089A (en) * | 2021-01-13 | 2021-05-04 | 上海幻电信息科技有限公司 | Test method and system for probability related function module |
| CN113010392B (en) * | 2021-02-08 | 2023-06-20 | 建信金融科技有限责任公司 | Big data platform testing method, device, equipment, storage medium and system |
| CN113010392A (en) * | 2021-02-08 | 2021-06-22 | 建信金融科技有限责任公司 | Testing method, device, equipment, storage medium and system for big data platform |
| CN112988424A (en) * | 2021-03-22 | 2021-06-18 | 广州虎牙科技有限公司 | Independent process access implementation method and device, electronic equipment and readable storage medium |
| CN112988424B (en) * | 2021-03-22 | 2024-02-27 | 广州虎牙科技有限公司 | Independent process access realization method and device, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110750458A (en) | Big data platform testing method and device, readable storage medium and electronic equipment | |
| CN109302522B (en) | Test method, test device, computer system, and computer medium | |
| US11755919B2 (en) | Analytics for an automated application testing platform | |
| US10372600B2 (en) | Systems and methods for automated web performance testing for cloud apps in use-case scenarios | |
| US20210311858A1 (en) | System and method for providing a test manager for use with a mainframe rehosting platform | |
| US10521322B2 (en) | Modeling and testing of interactions between components of a software system | |
| US9697104B2 (en) | End-to end tracing and logging | |
| CN111897724B (en) | Automatic testing method and device suitable for cloud platform | |
| CN110768872B (en) | Inspection method, system, device, computer equipment and storage medium | |
| US10339039B2 (en) | Virtual service interface | |
| US11256590B1 (en) | Agent profiler to monitor activities and performance of software agents | |
| US10942837B2 (en) | Analyzing time-series data in an automated application testing system | |
| CN112241360A (en) | Test case generation method, device, equipment and storage medium | |
| US8219856B2 (en) | Memory growth detection | |
| CN107807883B (en) | Unit test method and device for user mode network file system | |
| US10528456B2 (en) | Determining idle testing periods | |
| CN107168844B (en) | Performance monitoring method and device | |
| CN117370203B (en) | Automatic test method, system, electronic equipment and storage medium | |
| Wang et al. | A model-based framework for cloud API testing | |
| CN112650676A (en) | Software testing method, device, equipment and storage medium | |
| TW201629768A (en) | Infrastructure rule generation | |
| CN114064465A (en) | Stability testing method based on Linux cloud platform | |
| CN103176903B (en) | The test method and equipment of MapReduce distributed system program | |
| Seifermann | Application performance monitoring in microservice-based systems | |
| CN115221033A (en) | Interface protocol testing method and device, computer readable medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200204 |