CN111858365A - A method and equipment for Flink K-Means performance test - Google Patents

Abstract

The invention provides a method and equipment for testing the performance of Flink K-Means, wherein the method comprises the following steps: defining parameters needed by data generation based on a k-means clustering algorithm, analyzing the parameters and generating original data based on the analyzed parameters; carrying out format conversion on the original data to form a data format required by the test; testing the Flink K-Means based on the data after format conversion, and graphically displaying the test result; and analyzing specific parameters of the test result to judge whether the Flink distributed real-time processing engine can meet the current production requirement. By using the scheme of the invention, the performance of the Flink batch K-Means can be tested, each node of the Flink cluster can be tested, the test data result can be analyzed and counted to determine whether the Flink distributed real-time processing engine can meet the current production requirement, and whether the physical machine, the memory, the CPU and the hard disk can meet the on-line production requirement of the Flink.

Description

A method and equipment for Flink K-Means performance test

technical field

The art relates to the computer field, and more particularly to a method and device for Flink K-Means performance testing.

Background technique

With the rapid development of big data in recent years, many popular open source communities have emerged, among which are Hadoop, Storm, and later Spark, Flink, etc. With the rapid development of The mainstream choice of users.

Apache Flink is a distributed, high-performance, high-availability, and high-accuracy open source stream processing framework for data stream applications. The core of Flink is to provide data distribution, communication, and fault-tolerant distributed computing on data streams. At the same time, Flink provides batch stream fusion computing capabilities and SQL expression capabilities on the stream processing engine. The development of Flink technology is becoming more and more mature, and its PK trend with Spark is gradually gaining the upper hand. It is a hot new star in the current real-time processing field. Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed stream data flow engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner, and Flink's pipelined runtime system can execute batch and stream processing programs. In addition, Flink's runtime itself supports the execution of iterative algorithms. K-means clustering algorithm, also known as k-means clustering algorithm, is a simple and classic distance-based clustering algorithm. It uses the distance as the evaluation index of similarity, that is, the closer the distance between two objects, the greater the similarity. The algorithm considers that clusters are composed of objects that are close in distance, so it takes compact and independent clusters as the final goal.

Hibench is Intel's open source big data benchmarking tool that can evaluate the speed, throughput and system resource utilization of different big data frameworks. Including Sort, WordCount, TeraSort, Sleep, SQL, PageRank, Nutchindexing, Bayes, Kmeans, NWeight and enhanced DFSIO, etc. For the support of the Flink framework, currently only the test of streaming computing is supported, and the batch computing of the Flink distributed data stream engine is supported. mode, and cannot be tested, the present invention can realize efficient batch performance test of Flink K-Means.

SUMMARY OF THE INVENTION

In view of this, the purpose of the embodiments of the present invention is to propose a method and device for Flink K-Means performance test. By using the method of the present invention, the performance of Flink batch K-Means can be tested, and each node of the Flink cluster can be tested. The test can analyze and count the test data results to confirm whether the Flink distributed real-time processing engine can meet the current production requirements, and whether the physical machine, memory, CPU, and hard disk can meet the Flink online production requirements.

Based on the above purpose, an aspect of the embodiments of the present invention provides a method for Flink K-Means performance test, comprising the following steps:

Define the parameters required to generate data based on the k-means clustering algorithm, parse the parameters, and generate original data based on the parsed parameters;

Format the original data to form the data format required for testing;

Test Flink K-Means based on the converted data, and display the test results graphically;

The test results are analyzed with specific parameters to judge whether the Flink distributed real-time processing engine can meet the current production requirements.

According to an embodiment of the present invention, parameters required to generate data are defined based on a k-means clustering algorithm, and parsing the parameters and generating original data based on the parsed parameters includes:

select a point and add the point to the center set S;

Obtain the mean of all points in each dimension in the center set S, and calculate the new point by the mean of dimension + variance;

Add new points to the center set S, and loop the previous step until enough initial centers are obtained;

Generate points from the initial center through a Gaussian distribution around the initial center, divide the number of points that need to be generated, and write the data points generated for each partition into the result to generate the original data.

According to one embodiment of the invention, the parameters include file output location, dimension, number of data points, number of clusters, minimum distance from the mean of all centers, and standard deviation of the data points.

According to one embodiment of the present invention, the specific parameters include total throughput, average number of delays, error rate, and throughput, delay, and error rate for the same time interval on average.

According to an embodiment of the present invention, the data format includes a data volume of 500G, a number of threads of 100, and a k value of 3.

Another aspect of the embodiments of the present invention also provides a Flink K-Means performance test device, the device includes:

The parsing module, the parsing module is configured to define parameters required for generating data based on the k-means clustering algorithm, parse the parameters, and generate original data based on the parsed parameters;

A conversion module, the conversion module is configured to format the original data to form a data format required by the test;

Test module, the test module is configured to test Flink K-Means based on the format-converted data, and display the test results graphically;

Analysis module, the analysis module is configured to analyze the test results with specific parameters to judge whether the Flink distributed real-time processing engine can meet the current production requirements.

According to an embodiment of the present invention, the parsing module is further configured to:

select a point and add the point to the center set S;

Obtain the mean of all points in each dimension in the center set S, and calculate the new point by the mean of dimension + variance;

Add new points to the center set S, and loop the previous step until enough initial centers are obtained;

Generate points from the initial center through a Gaussian distribution around the initial center, divide the number of points that need to be generated, and write the data points generated for each partition into the result to generate the original data.

According to one embodiment of the invention, the parameters include file output location, dimension, number of data points, number of clusters, minimum distance from the mean of all centers, and standard deviation of the data points.

According to one embodiment of the present invention, the specific parameters include total throughput, average number of delays, error rate, and throughput, delay, and error rate for the same time interval on average.

According to an embodiment of the present invention, the data format includes a data volume of 500G, a number of threads of 100, and a k value of 3.

The present invention has the following beneficial technical effects: the method for Flink K-Means performance test provided by the embodiment of the present invention defines parameters required for generating data based on the k-means clustering algorithm, parses the parameters, and generates original data based on the parsed parameters; The original data is formatted to form the data format required for the test; Flink K-Means is tested based on the format-converted data, and the test results are displayed graphically; the test results are analyzed with specific parameters to judge Flink distributed real-time The technical solution of whether the processing engine can meet the current production requirements, can test the performance of Flink batch K-Means, can test each node of the Flink cluster, and can analyze and count the test data results to confirm the Flink distributed real-time processing engine Whether it can meet the current production needs, and whether the physical machine, memory, CPU, and hard disk can meet the Flink online production needs.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can also be obtained according to these drawings without creative efforts.

1 is a schematic flow chart of a method for Flink K-Means performance testing according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a device for performing a Flink K-Means performance test according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention will be further described in detail below with reference to the specific embodiments and the accompanying drawings.

Based on the above purpose, in the first aspect of the embodiments of the present invention, an embodiment of a method for testing the performance of Flink K-Means is proposed. Figure 1 shows a schematic flow chart of the method.

As shown in Figure 1, the method may include the following steps:

S1 defines the parameters required to generate data based on the k-means clustering algorithm, parses the parameters, and generates original data based on the parsed parameters;

S2 converts the original data into a format to form the data format required for the test;

S3 tests Flink K-Means based on the format-converted data, and displays the test results graphically;

S4 analyzes the test results with specific parameters to judge whether the Flink distributed real-time processing engine can meet the current production requirements.

Through the technical solution of the present invention, the performance of Flink batch K-Means can be tested, each node of the Flink cluster can be tested, and the test data results can be analyzed and counted to confirm whether the Flink distributed real-time processing engine can meet the current production requirements. Demand, whether the physical machine, memory, CPU, and hard disk can meet the Flink online production requirements.

In a preferred embodiment of the present invention, the parameters required to generate data are defined based on the k-means clustering algorithm, and parsing the parameters and generating the original data based on the parsed parameters includes:

select a point and add the point to the center set S;

Obtain the mean of all points in each dimension in the center set S, and calculate the new point by the mean of dimension + variance;

Add new points to the center set S, and loop the previous step until enough initial centers are obtained;

Generate points from the initial center through a Gaussian distribution around the initial center, divide the number of points that need to be generated, and write the data points generated for each partition into the result to generate the original data.

The Flink K-Means data generation module flink-data-generator is developed based on the java language. The core of this module is a distributed K-means data generator. It uses java's IO streaming programming and the original K-means algorithm to realize the function of distributed generation of simulated data. You need to compile the module source code with mvn clean package to generate data-generator.jar, and copy the jar package to ${FLINK_HOME}/example/batch.

Specifically, the definition module receives parameters such as output (file output location), dimension, number of data points, number of clusters, minimum distance from the mean of all centers, standard deviation of data points and other parameters, parses these parameters and initializes them. Then pick a point and add it to the center set S and create the next point, get the mean of all points in each dimension in S, the new point will be calculated by the mean of the dimension + variance, variance = minDistance + (minDistance*rnd.nextGaussain ), add the calculated new points to S, and loop from the previous step until enough initial centers are obtained, generate points around the center through Gaussian distribution from the initial centers, divide the number of points that need to be generated, and generate data for each partition Click to write the result. The generated data mainly uses classes such as BufferedWriter FileWriter of java io module, and Random class of util. Then convert the data format, obtain the data generated in the previous steps as input data, format the data, use Flink apiDataSet to define the data output "-output", the file receiver is lazy, and the output position is triggered by parameters.

In a preferred embodiment of the present invention, the parameters include file output location, dimension, number of data points, number of clusters, minimum distance from the mean of all centers, and standard deviation of data points.

In a preferred embodiment of the present invention, the specific parameters include the total throughput, the average number of delays, the error rate, and the throughput, delay, and error rate at the same time on average. Judging whether the Flink distributed real-time processing engine can meet the current production needs based on specific parameters includes:

1. During the execution of the test task, the test data analysis module calls the Flink Restfull web interface GET/v1/jobs/<jobid>, and calculates the throughput (AvgQPS/TPS (bar/ minutes)), delay, the total number of errors, record the execution time at the end of the test, and output the record in real time, the queue service is used for a large amount of data;

2. According to the total number of data and execution time, the method of tail-cutting mean is used to calculate the average number of total throughput, delay, and error rate, as well as the throughput, delay, and error rate at the same time of average interruption;

3. At the end of the test, output to the excel file through the interface in a certain format, that is, the single test result. According to the test result, it is judged whether the Flink distributed real-time processing engine can meet the current production requirements.

In a preferred embodiment of the present invention, the data format includes a data volume of 500G, a number of threads of 100, and a k value of 3. Using data-generator.jar to generate simulated data, you can customize the amount of data, the number of threads, and set the generated location, number of bars, k value, etc., which can generate test data accurately and efficiently.

Flink run -c DistributedDataGenerator data-generator-1.0-SNAPSHOT.jar--output hdfs://xx.xx.xx.xx:9000/flink/kmeans--d 100--size500000000--k 3

Among them: Flink command, the data volume is 500G, the number of threads is 100, and the k value is 3.

Finally, the data is output to the hadoop system to store hdfs://xx.xx.xx.xx:9000/flink/kmeans. The output method uses the interface call, the interface development conforms to the RESTfull interface specification, and this function is encapsulated into a module, which can be automated Implement data output.

In a preferred embodiment of the present invention, testing Flink K-Means based on the converted data includes: running the Flink K-Means test module, using the K-Means application program that comes with ${FLINK_HOME}/example/batch CLI: KMeans.jar, and store the test results to the distributed system Hadoop.

flink run KMeans.jar--points hdfs://xx.xx.xx.xx:9000/flink/kmeans--output hdfs://xx.xx.xx.xx:9000/flink/kmeans/Output--k 3--iterations 20

Among them: Flink command, the data input source is the data generated and stored in Hadoop in step 1, the result data output is stored in Hadoop, iterations is the number of iterations.

In a preferred embodiment of the present invention, displaying the test results graphically includes:

1. Flink configures metrics monitoring to detect the Flink test process,

metrics.scope.jm,metrics.scope.jm.job,

metrics.scope.tm, metrics.scope.tm.job, etc.

The above is Flink's configuration items about metrics, which can monitor and collect flink's job, task and other indicator data;

2. Flink configuration pushes monitoring information to Prometheus,

metrics.reporter.promgateway.class, metrics.reporter.promgateway.host, etc. The above are the configuration items of Flink about metrics, and configure the Prometheus service information;

3. Configure Prometheus, download and install Prometheus components, and configure Prometheus;

4. The Flink service calls the built-in interface metrics to collect monitoring information and calls the Prometheus Restfullweb interface to push the monitoring data to Prometheus in real time. Through the Prometheus UI, you can see the monitoring results such as throughput and delay in real time graphically.

Through the technical solution of the present invention, the performance of Flink batch K-Means can be tested, each node of the Flink cluster can be tested, and the test data results can be analyzed and counted to confirm whether the Flink distributed real-time processing engine can meet the current production requirements. Demand, whether the physical machine, memory, CPU, and hard disk can meet the Flink online production requirements.

It should be noted that those of ordinary skill in the art can understand that all or part of the process in the method of the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it may include the flow of the embodiments of the above-mentioned methods. The storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like. The above computer program embodiments can achieve the same or similar effects as any of the foregoing method embodiments corresponding thereto.

In addition, the methods disclosed according to the embodiments of the present invention may also be implemented as a computer program executed by the CPU, and the computer program may be stored in a computer-readable storage medium. When the computer program is executed by the CPU, the above-mentioned functions defined in the methods disclosed in the embodiments of the present invention are executed.

Based on the above purpose, in the second aspect of the embodiments of the present invention, a device for Flink K-Means performance test is proposed. As shown in FIG. 2 , the device 200 includes:

The parsing module, the parsing module is configured to define parameters required for generating data based on the k-means clustering algorithm, parse the parameters, and generate original data based on the parsed parameters;

A conversion module, the conversion module is configured to format the original data to form a data format required by the test;

Test module, the test module is configured to test Flink K-Means based on the format-converted data, and display the test results graphically;

Analysis module, the analysis module is configured to analyze the test results with specific parameters to judge whether the Flink distributed real-time processing engine can meet the current production requirements.

In a preferred embodiment of the present invention, the parsing module is further configured to:

select a point and add the point to the center set S;

Obtain the mean of all points in each dimension in the center set S, and calculate the new point by the mean of dimension + variance;

Add new points to the center set S, and loop the previous step until enough initial centers are obtained;

Generate points from the initial center through a Gaussian distribution around the initial center, divide the number of points that need to be generated, and write the data points generated from each partition to the result to generate the original data.

In a preferred embodiment of the present invention, the parameters include file output location, dimension, number of data points, number of clusters, minimum distance from the mean of all centers, and standard deviation of data points.

In a preferred embodiment of the present invention, the specific parameters include the total throughput, the average number of delays, the error rate, and the throughput, delay, and error rate at the same time on average.

In a preferred embodiment of the present invention, the data format includes a data volume of 500G, a number of threads of 100, and a k value of 3.

It should be particularly pointed out that the embodiments of the above system use the embodiments of the above method to specifically describe the working process of each module, and those skilled in the art can easily think of applying these modules to other embodiments of the above method.

In addition, the above-mentioned method steps and system units or modules can also be implemented by using a controller and a computer-readable storage medium for storing a computer program that enables the controller to implement the functions of the above-mentioned steps or units or modules.

Those skilled in the art will also appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the functions in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.

The above-described embodiments, particularly any "preferred" embodiments, are possible examples of implementations, and are presented merely for a clear understanding of the principles of the invention. Numerous changes and modifications may be made to the above-described embodiments without departing from the spirit and principles of the technology described herein. All modifications are intended to be included within the scope of this disclosure and protected by the appended claims.

Claims (10)

1. a method for Flink K-Means performance test, is characterized in that, comprises the following steps: Define parameters required to generate data based on the k-means clustering algorithm, parse the parameters, and generate original data based on the parsed parameters; performing format conversion on the raw data to form a data format required by the test; Test the Flink K-Means based on the format-converted data, and display the test results graphically; The test results are analyzed with specific parameters to determine whether the Flink distributed real-time processing engine can meet the current production requirements. 2. The method according to claim 1, wherein the parameters required for generating data are defined based on a k-means clustering algorithm, and analyzing the parameters and generating raw data based on the analyzed parameters comprises: select a point and add said point to the center set S; Obtain the mean value of all points of each dimension in the center set S, and calculate the new point by the mean value of dimension + variance; Add the new point to the center set S, and loop the previous step until enough initial centers are obtained; From the initial center, points are generated around the initial center through a Gaussian distribution, the number of points to be generated is divided, and the data points generated by each partition are written into the result to generate the original data. 3. The method of claim 1, wherein the parameters include file output location, dimension, number of data points, number of clusters, minimum distance from the mean of all centers, and standard deviation of data points. 4 . The method according to claim 1 , wherein the specific parameters include total throughput, average number of delays, error rate, and throughput, delay, and error rate at the same time of average interruption. 5 . 5 . The method according to claim 1 , wherein the data format includes a data volume of 500G, a number of threads of 100, and a k value of 3. 6 . 6. A device for Flink K-Means performance test, wherein the device comprises: a parsing module, the parsing module is configured to define parameters required for generating data based on a k-means clustering algorithm, parse the parameters, and generate raw data based on the parsed parameters; a conversion module, the conversion module is configured to perform format conversion on the original data to form a data format required by the test; A test module, which is configured to test the Flink K-Means based on the format-converted data, and display the test results graphically; An analysis module, which is configured to analyze the test results with specific parameters to determine whether the Flink distributed real-time processing engine can meet the current production requirements. 7. The device according to claim 6, wherein the parsing module is further configured to: select a point and add said point to the center set S; Obtain the mean value of all points of each dimension in the center set S, and calculate the new point by the mean value of dimension+variance; Add the new point to the center set S, and loop the previous step until enough initial centers are obtained; From the initial center, points are generated around the initial center through a Gaussian distribution, the number of points to be generated is divided, and the data points generated by each partition are written into the result to generate the original data. 8. The apparatus of claim 6, wherein the parameters include file output location, dimension, number of data points, number of clusters, minimum distance from the mean of all centers, and standard deviation of data points. 9 . The device according to claim 6 , wherein the specific parameters include total throughput, average number of delays, error rates, and throughputs, delays, and error rates with an average interruption at the same time. 10 . 10 . The device according to claim 6 , wherein the data format includes a data volume of 500G, a number of threads of 100, and a k value of 3. 11 .

Images