CN108683560B - Performance benchmark test system and method for large data stream processing framework - Google Patents

Abstract

The invention relates to a performance benchmark test system and a method of a large data flow processing framework. The method generates the load according with the data characteristics of the streaming processing mode by selecting the application according with the calculation characteristics of the streaming processing mode, tests the performance of the large data stream processing frame under typical scenes and applications, collects performance indexes such as back pressure, throughput, delay, system resources, node data and the like during operation, and finally diagnoses the bottleneck of the streaming processing frame by analyzing and counting the collected data.

Description

Performance benchmark test system and method for large data stream processing framework

Technical Field

The invention relates to a performance benchmark test system and method for a large data stream processing framework, in particular to performance expression during the operation of the framework under a typical stream scene and an application, and belongs to the technical field of software.

Background

With the coming of the internet era and the continuous development of technologies such as mobile internet, social networks, electronic commerce and the like, data shows explosive growth, and big data becomes a hot spot of current scientific and technological circles, business circles and even government concerns.

In general, data can be divided into bounded data and infinite data. Bounded data, also called batch data, refers to data stored in persistent media that is fixedly bounded, and the amount of data does not change during calculation. Generally, a batch big data processing framework (hereinafter referred to as a batch processing framework) receives tasks submitted by users to perform logic processing and analysis on stored data sets, and finally outputs results. For example, a machine learning algorithm is used for analyzing and mining the historical data set to build a prediction model. Many mature batch processing frameworks have been used today, such as Hadoop, Spark, etc.

However, with the rise or wide application of sensing devices and social networks, the demand for real-time analysis of massive high-speed data is increasing, and such continuously generated and endless data is called endless data, also called streaming data. An investigation of enterprise informatization by foreign consulting agencies has shown that 70% of enterprises have a need for real-time processing of streaming data (Liu X, Iftikhar N, Xie X. surface of real-time processing systems for big data. International Database Engineering & application Symposium ACM. New York. USA 2014: 356-361). For example, the Alibama updates a commodity search engine in real time based on a Blink framework, and an online machine learning platform is constructed; the Mei-Tuan network is based on a Storm framework, user behaviors are analyzed, and quasi-real-time recommendation feedback is achieved; the drip-out line monitors the generation place of order data and draws a geographical thermodynamic diagram early warning based on a Samza framework.

However, the stream data has characteristics different from those of the batch data, and the traditional batch processing framework cannot process the stream data well, so that a stream big data processing framework (hereinafter, referred to as a stream processing framework) is generated. Although the stream processing framework is still in the development stage, the stream processing framework has become a focus of attention in academic and industrial fields as the stream processing scenario becomes more important. The mainstream flow processing framework today is Storm, Flink, etc.

The cluster environment is larger and larger, the probability of system performance problems is increased, and the nodes can have the problems of failure, insufficient resources and the like in unpredictable time or data (grand, Zhang Guangby, Zheng Yun Min. large dataflow type calculation: key technology and system example [ J ]. software academic, 2014, (04): 839-. In a streaming processing scene, the system throughput may be reduced and the delay may be increased due to the reasons of too large load, unreasonable parameter configuration, and the like; the node processing rate is not as fast as the input rate, and the back pressure phenomenon may occur; data is unevenly distributed, which may lead to a single point of resource bottleneck. The requirement of the real-time performance of the streaming processing is strict, and the tolerance of a user is low, so that the stability of the system performance in the streaming processing is especially important. However, in the current solution to the performance problem of the big data system, after the problem occurs, if a scene and an application which may have the performance problem can be constructed in advance, and a test is performed in an actual production cluster, the problem of resources or configuration in the system can be found in advance, and the loss in actual operation is reduced.

The flow processing framework has only begun to evolve in recent years, and therefore, performance benchmarking for flow processing frameworks is not a well established, uniform standard within the industry and is small in number. Yahoo Streaming benchmarking Benchmarks (Yahoo training benchmarking https:// githu. com/Yahoo/Streaming-marks) are the Streaming framework test Benchmarks designed by the Yahoo company. The method comprises the steps of generating data by Kafka, selecting a stream processing framework to be tested to execute, and interacting with an external Redis database. But the test benchmark only provides a test application of the filtering operation, and the overall completeness is low. The stream bench (Lu R, Wu G, Xie B, et al. stream bench: Towards benchmarking modified distributed stream Computing frames [ C ]// Utility and Cloud Computing (UCC),2014IEEE/ACM 7th International Conference on. IEEE,2014:69-78.) is also a test benchmark developed for stream processing frameworks that includes an application set of 7 filtering or statistical operations to test the delay, throughput and fault recovery capabilities of the stream processing framework. However, the reference does not support complex test applications such as windows, and also does not support dynamically changing data sources. In addition, some of the papers work in this field simply relate to Chintalli S et al (Chintalli S, Dagit D, Evans B, et al, benchmark Streaming applications: Storm, flash and Spark Streaming [ C ]// Parallel and Distributed Processing Symposium works, 2016IEEE International, IEEE,2016:1789-1792.) to compare the performance of Spark Streaming, Storm and Flink stream Processing frameworks, a simple application was designed to enter data with Kafka for filtering, linking, and aggregation operations; two test applications, window aggregation and window joining, were constructed by Karimov J et al (Karimov J, Rabl T, Katsifodimios A, et al. However, these works all have problems such as too small coverage of the application test set.

In summary, the existing performance benchmark test for the flow processing framework has three disadvantages in terms of application test sets, flow data sources and performance indexes, one is that the dynamically-changing data sources are not supported, the application structure is too simple, the coverage degree of the flow processing characteristics is low, the majority of the performance indexes only consider delay and throughput, and no other indexes such as backpressure are involved.

Disclosure of Invention

The technical problem of the invention is solved: the system and the method overcome the defects of the prior art, and particularly aim at the characteristics of a streaming processing mode, construct a data source covering the characteristics so as to test the performance of the framework in a typical streaming scene and analyze and diagnose the bottleneck of the streaming processing framework.

The technical scheme of the invention relates to a performance benchmark test system of a large data stream processing framework, which comprises a stream type load generator, a stream type scene and application constructor, a performance data acquisition tool and a performance data analysis tool.

And the streaming load generator is responsible for generating the load according with the characteristics of the streaming processing mode data. Unlike traditional batch data generation, the streaming load generator includes both the design of flow rate patterns and the design of dataset properties. The flow rate pattern refers to a pattern of change in the frame input rate over time, the rate being either a constant value or a changing value, or a change that follows a function. Dataset properties refer to characteristics of the dataset flowing in per second, including dimensionality, degree of misordering, degree of tilt, and the like. By combining the flow rate pattern and the dataset properties, the generation of a streaming load can be achieved.

The flow type scene and application constructor is responsible for constructing scenes and applications covering flow data calculation characteristics, and the scenes and the applications of the test system mainly come from two aspects, namely flow data processing scenes frequently encountered in real life and test cases provided by a current flow processing framework test benchmark. Meanwhile, the constructor especially considers a window mechanism in the streaming processing, and performs parameter test of the control variable by changing different window influence parameter values.

The performance data acquisition tool is responsible for acquiring various performance indexes of test application in the test process, wherein the indexes comprise some finer-grained node information such as the processing rate, the processed data volume, the buffer pool usage amount and the like of each node of a flow processing frame in the test besides the throughput rate, the delay, the back pressure and the system resources.

And the performance data analysis tool is responsible for processing the acquired data, visualizing the data into a chart according to a top-down analysis statistical method, reflecting the influence of parameter change on indexes such as throughput, delay, back pressure, system performance and the like, and analyzing the bottleneck of a flow processing frame due to diagnosis.

The invention also relates to a performance benchmark test method of the large data flow processing framework, which comprises the following implementation steps:

1) and deploying the test cluster, and determining configuration parameters of the flow processing framework to be tested, wherein the configuration parameters comprise the memory, the CPU, the Slave number, the maximum cluster parallelism and the like.

2) Selecting a flow scene and application to be tested, setting a flow rate mode and data set attributes, and configuring required parameters. The test parameters are mainly divided into two categories, namely a data source module comprising flow rate, data gradient, disorder degree and the like; and the second is window module parameters including window type, parallelism, window size and the like. And then, the tester determines a parameter value to be changed, keeps other environment configurations unchanged, and performs a plurality of groups of tests on the cluster.

3) And running a test script corresponding to the application, and starting a performance data acquisition tool at the same time.

4) Waiting for the test to complete. In the test process, the performance data acquisition tool periodically accesses the cluster to acquire data and persists the data in the hard disk after the test is finished.

5) When the test application is completed or finished on the cluster, analyzing performance data acquired during running, comparing the influence of the change of the parameter variable on throughput, delay, back pressure and system resources, analyzing layer by layer from top to bottom, and positioning bottleneck points.

Compared with the existing big data processing frame test standard, the invention has the following advantages:

(1) compared with the existing big data test benchmark, the method generates the streaming load which accords with the actual dynamic change. In the aspect of flow rate mode, four different flow rate generation modes are designed; in terms of dataset attributes, the generation of out-of-order, skewed datasets is considered.

(2) Compared with the existing big data test benchmark, a scene and application covering the characteristics of the streaming processing mode are constructed, particularly a window mode in streaming data processing, and the window type and parameters influencing the window are comprehensively considered.

(3) Compared with the existing big data test standard, the method not only considers the general throughput and delay performance indexes, but also collects and analyzes the specific back pressure index and node data in the flow processing frame, and simultaneously considers the system resources of the cluster.

Drawings

FIG. 1 is a diagram of the performance benchmarking system architecture of the present invention;

FIG. 2 is a flow chart of window selection according to the present invention;

FIG. 3 is a graph of performance comparison results for ProductStatis applications at different gradients; the left graph is a backpressure comparison graph under different gradients, and the right graph is a graph of actual output rates of the data source end under different gradients;

FIG. 4 is a schematic diagram of the inclination influence data distribution;

FIG. 5 is a graph of performance versus results for TransactionJoin applications with different window sizes; the left graph is a back pressure comparison graph under different window sizes, and the right graph is a delay comparison graph under different window sizes;

FIG. 6 is a diagram of a window size impact computation quantity diagram; the left graph is the data calculated amount under a small window, and the right graph is the data calculated amount under a large window;

FIG. 7 is a graph comparing the performance of the ProductStatis and TransactionJoin applications at different rated rates. The upper left graph is a graph of inverse pressure comparison of the PortactStatis application at different rated speeds, the upper right graph is a graph of actual output speed comparison of the PortactStatis application at different rated speeds, the lower left graph is a graph of inverse pressure comparison of the TransactionJoin application at different rated speeds, and the lower right graph is a graph of actual output speed comparison of the TransactionJoin application at different rated speeds.

Detailed Description

The invention is described in detail below with reference to the figures and specific examples.

The invention provides a performance benchmark test system and a performance benchmark test method for a large data stream processing framework.

Streaming processing modes have their own unique features, which are herein divided into both data and computational features. The data characteristics mainly refer to characteristics of streaming data to be processed by a framework, and include the following five types:

1) and (4) real-time performance. Stream data is generated in real time, arrives in real time, and needs to be processed in real time.

2) And (4) time sequence. The stream data has sequential logic, which is time dependent. However, the data source is not unique, the arrival sequence of each tuple in the data stream is independent, and time disorder may occur during processing.

3) And is not limited. Stream data is in units of tuples, data is infinite and continuously generated.

4) Dynamic variability. The rate and distribution of the streaming data are affected by the actual production environment at the current moment and cannot be predicted in advance.

5) Difficult to reproduce. Once the data is processed, it cannot be retrieved again, or is expensive to retrieve again, unless purposely saved.

In addition, the calculation features of the streaming processing mode can be specifically classified into the following four types:

1) real-time performance of the calculation. In the flow processing framework, the data sent from the data source end flow among operators, and the result is obtained after calculation and analysis. Stream data is generated in real time and arrives in real time, real-time processing is needed, and the frame has higher low-delay requirements.

2) Ordering of the calculations. The flow data and the time are closely related, but the data arrival sequence is unpredictable, and due to network delay and the like, the out-of-order data flow is inevitably generated when an operator receives the data, so a time-oriented processing strategy is needed to perform ordered calculation on the out-of-order data, and a water line mechanism (Watermark) is a common out-of-order processing strategy in a flow processing framework.

3) The boundedness of the calculation. Streaming data continues to be produced, which places new demands on the semantics of aggregation, concatenation, etc., that otherwise operate on a bounded set of data, the limitless nature of the data making it impossible to wait until all data reception is complete before processing. Thus, streaming processing typically employs a windowing scheme, whereby bounded sets are partitioned over an infinite stream of data based on some rule, and then a dependent aggregation or join operation is performed on the bounded sets. Window mode is also a large feature of streaming processes as opposed to batch processes.

4) High reliability of operation. The batch processing calculation is persistent offline data, and if problems occur in the operation, the calculation is simply and repeatedly loaded. However, the repeated calculation of stream data is expensive, which requires that the streaming process can quickly recover the cluster state at a certain moment after the failure occurs, so as to avoid the repeated calculation. The stream processing framework failure recovery semantic guarantees are divided into three types, namely at most once, at least once and exactly once, and different stream processing framework failure recovery semantic guarantees are different.

Currently, popular Streaming processing applications are all running on a distributed Streaming processing framework, and the distributed Streaming processing framework in the mainstream in the industry is Storm, Spark Streaming, Flink, and the like. Flink is an open-source distributed stream processing framework, adopts a continuous stream processing mode, has the advantages of high throughput and low delay, and supports 'exactly once' failure recovery semantics. Compared with other stream processing frameworks, Flink designs a perfect processing mechanism aiming at the characteristics of a stream processing mode, and simultaneously considers batch processing as a special case of stream processing and unifies stream batch at the bottom layer of the framework. Therefore, Flink is herein chosen as the target framework to be tested.

The application architecture used in the embodiment of the present invention is shown in fig. 1, the upper half is a logic diagram of the application to be tested when running in the Flink stream processing framework, and the lower half is a module architecture of the system of the present invention (stream processing framework benchmark test system), and the module architecture includes four component modules, namely a stream load generator, a scenario and application test set, a performance data acquisition tool, and a performance data analysis tool.

(1) Streaming load generator

Responsible for generating streaming load that conforms to the characteristics of the streaming mode data. The streaming load generator includes a design of both flow rate patterns and data set attributes, based on the data characteristics.

In the flow rate mode, the invention designs four different flow rate modes: fixed rate, i.e., the amount of data generated by the data source per second is constant; random rate, i.e., the amount of data generated by the data source per second, is taken randomly within a range; the burst rate, i.e. the data source suddenly increases from a stable smaller value to the limit rate in a short time; the exponential rate, i.e., the amount of data generated by the data source per second, grows according to an exponential function and cycles through the period. Three different dataset attributes are specified simultaneously: the data dimension is larger, the size of each piece of data is larger, and the network transmission quantity is influenced; the data gradient describes the gradient degree of Key distribution for Key/Value form data, and the higher the gradient is, the more uneven the Key distribution is; and the disorder degree describes the disorder degree of the time of the input data stream, and the greater the disorder degree is, the more serious the disorder degree of the time of some data in the input data stream is.

(2) Streaming scene and application constructor

A test set containing a performance benchmark test system. On the one hand, the test set needs to construct comprehensive scenes and applications to cover the computation characteristics of the streaming processing mode, and on the other hand, the applications need to be capable of fitting the processing logic in actual production life, and on the basis of the two points, the test set designs five scenes and eight test applications typical for streaming processing, as shown in table 1.

TABLE 1 introduction of scenarios and exemplary applications

Table 2 compares the feature coverage of the application set involved in the performance benchmarking system and other benchmarks or related paper work, the system covers a wider range of both window types and operator operations than other benchmarks.

TABLE 2 comparison of feature coverage for each test benchmark application set

Among them, the window processing mechanism is a big feature of the stream processing framework, and the parameters of this part are the most complex. There are many parameters that affect the window, such as window size, sliding step size, operator parallelism, trigger computation function, etc., some of which are independent of each other and some of which are dependent on each other. In the window implementation of the application, the benchmark test system interfaces different windows through different parameter configurations, and the selection process is as shown in fig. 2.

1) Firstly, determining whether a window is a Key-based window, if so, carrying out partition operation on a keyby function before carrying out window operator on a data stream; if the window is not a common window based on Key, the parallelism of the window operator can only be set to 1, and parallel computation cannot be carried out.

2) Then, the driving mode of the window is determined and divided into a counting driving window and a time driving window. The counting window triggers the window to calculate through the number of data in the window, and the time window triggers the window to calculate through setting time. The time window has the difference between the occurrence time (Event-time) and the Processing time (Processing-time), and if the occurrence time is adopted, a water level line operator and a maximum allowed disorder time parameter are required to be set to process the disorder condition.

3) Then, the moving type of the window is determined, and the window is divided into a rolling window, a sliding window and a session window. The size of the rolling window only needs to be set, the size of the sliding window and the sliding step length need to be set, the conversation window only belongs to the time driving window, and the inactive time interval needs to be set.

4) Finally, the access window calculates a function, which is included in the application logic and is not passed as a parameter.

(3) Performance data acquisition tool

And the system is responsible for collecting various performance indexes of the test application, wherein the indexes comprise more fine-grained information such as throughput rate, delay, back pressure, system resource, speed of each node during operation, processing data volume, buffer pool usage and the like shown in table 3. The invention uses the Profiler acquisition tool to acquire data, and periodically accesses the Master node of the cluster to acquire the runtime data of the test application at the current moment. And after the test is finished, the Profiler persists all data of the test to a disk according to the acquisition time and stores the data in a Json format.

TABLE 3 Performance index

(4) Performance data analysis tool

And the device is responsible for carrying out statistical analysis on the collected performance data. The method comprises two stages, wherein in the first stage, the statistics and analysis are carried out on the persistent Json format data, and the multiple test results are jointly compared and converted into an intermediate result in a Csv format; the second stage converts the Csv format data into visualized chart data. And drawing a back pressure graph, a delay graph, a throughput rate graph and the like according to the intermediate result stored in the Csv file, reflecting the influence of parameter change on the performance, and diagnosing the bottleneck of the flow processing frame.

Apache Flink is used as a target frame to be tested, eight constructed scenes are used for testing on the cluster, and performance problems of the Flink in various typical streaming processing scenes are discovered and summarized.

(1) Effect of Tilt on Performance

Fig. 3 is a graph of performance comparison results of the product statis application under different gradients, where the left graph is a backpressure comparison graph under different gradients, the right graph is a graph of actual output rate of the data source under different gradients, and the values of the data gradients are {0,1,4} which respectively represent no gradient, low gradient, and high gradient. From the left graph, it can be derived that under the same other configuration parameters, a HIGH level of back pressure occurs at HIGH inclination, and a LOW level of back pressure occurs at no inclination and LOW inclination; the right graph shows that the data volume output by the Source terminal is close to 4000K/s in the low-tilt and no-tilt tests, but the data volume generated by the Source terminal is less than 2000K/s in the high-tilt test.

By analysis, the following conclusions were drawn: as shown in fig. 4, the processing of the window operator is related to the keys, and data of the same Key will be distributed to the same stream for computation. In the figure, data of Key 1 and Key 3 are assigned to the node of the window (1/2), and Key 2 and Key 4 are assigned to the node of the window (2/2). In the no-tilt and LOW-tilt tests, the data amount of each window node is equal or the difference is not large, and the reason that the backpressure level is LOW is mainly the influence of the data source end rate. In the HIGH-inclination test, the occupation ratio of one or more keys in a data set is HIGH, so that the data volume obtained by a certain window node is very large, the execution time of window calculation is increased, the processing rate of the node is lower than the data input rate at the moment, data which cannot be processed in time is backlogged in a buffer area of an input end, and after the buffer area of the input end is full, blocking and HIGH level back pressure occur. In order to slow down or stop the buffer growth, the system feeds back the blocking condition to the upstream node, so that the upstream sending amount is reduced, the processing rate is slowed, and this feeds back to the upstream … … so as to perform the step-by-step feedback, and finally the Source end output data amount is reduced.

(2) Effect of Window size on Performance

FIG. 5 is a graph showing the comparison of performance of TransactionJoin using different window sizes, wherein the left graph shows the comparison of backpressure for different window sizes, and the right graph shows the comparison of delay for different window sizes. The left graph can be obtained, and the back pressure level is higher along with the increase of the window size in the window operator; the right graph is the maximum value of the delay collected at the output end during test execution, and the delay has two sources due to the two data sources, and the transverse comparison shows that the delay difference between the two data sources is not large. But as the window size in the window operator increases, the delay received at the output increases.

The analysis leads to the conclusion that: as shown in fig. 6, the larger the window contains more data, and the calculation triggered by the window operator is a complex connected calculation whose execution time increases with the square level of the data, so that the processing rate of the window node is smaller than the input rate when the calculation is triggered, the data is backlogged in the buffer at the input end, and the back pressure occurs when the buffer at the input end is full. After the data is fed back to the data source, the data input of the whole system is reduced, the time consumption of calculation is long, the waiting time of the data in the nodes is increased, and the delay is increased.

(3) Effect of Rate on Performance

FIG. 7 is a comparison of the performance of the ProductStatis and TransactionJoin applications at different rated rates. The upper left is a comparison graph of the backpressure of the PortuctStatis application under different rated speeds, the upper right is a comparison graph of the actual output speed of the PortuctStatis application under different rated speeds, the lower left is a comparison graph of the backpressure of the TransactionJoin application under different rated speeds, and the lower right is a comparison graph of the actual output speed of the TransactionJoin application under different rated speeds. In the results of the ProductStatis application, the back pressure changes from OK to LOW as the rated rate increases. The change of the rate generated by the actual data source end with time is in accordance with the expectation under the rated rates of 10k/s and 160k/s, but the expectation is not reached under the rated rates of 640k/s and 1000k/s, and the actual rates of the two are close, which indicates that the system rate increases and a bottleneck occurs, but no HIGH level back pressure occurs at the time, and the reason for judging that the bottleneck occurs in the rate is the network bandwidth. In the results of the TransactionJoin application, a HIGH level of back pressure was present at all rated rates except for the 1k/s rate at which the back pressure level was OK. The actual data output rate of the data source end shows that the other actual rates can not reach the rated value except the rated rate of 1 k/s. The two applications are compared transversely to judge that the complexity of the application computing logic influences the back pressure of the system at the rated speed.

The productstatus application does not reach the rated rate, but the rate is always maintained stable, the actual rate of the TransactionJoin application fluctuates greatly with time, and the rate is in a periodic cycle of rising for a period of time, falling back quickly, and then rising again. The method includes the steps that the reason of rate fluctuation is explored, for the computation application of complex logic windows such as TransactionJoin and the like, when the windows do not trigger computation, the input end always receives data, and no back pressure occurs; when the window is calculated, because the calculation time is long, the processing rate is not as high as the receiving rate, the data backlog of the buffer area generates high back pressure, and the actual output rate of the current data source end is reduced due to feedback; after the window is calculated, the data reception of the backlog in the buffer area is recovered, the backpressure level is reduced, and the speed is increased; when the window calculation is triggered again, the data in the buffer area starts to be backlogged again, so that the back pressure and the speed rate show the inverse relationship of fluctuation. For the simple calculation application of the ProductStatis, the execution time required by window trigger calculation is short, the buffer area is not full in the calculation process, and at the moment, the balance state can be basically maintained as long as the input rate is reduced.

Although specific embodiments of the invention have been disclosed for illustrative purposes and the accompanying drawings, which are included to provide a further understanding of the invention and are incorporated by reference, those skilled in the art will appreciate that: the corresponding methods and tools may be implemented on other platforms without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the embodiment and the drawings.

Claims (2)

1. A performance benchmarking system for large data flow processing frameworks, comprising: the system comprises a streaming load generator, a streaming scene and application constructor, a performance data acquisition tool and a performance data analysis tool;

the streaming load generator generates streaming data containing data parameters; the flow load generator comprises a flow rate mode and a data set attribute, wherein the flow rate mode refers to a change mode of a frame input rate along with time, and the rate is a constant value or a change according to a certain function; dataset properties refer to characteristics of the dataset flowing in per second, including dimensionality, degree of misordering, and degree of tilt; the generation of the streaming load is realized by combining the flow rate mode and the data set attribute; the data parameters include flow load characteristics including flow rate patterns or data set attributes, the flow rate patterns including fixed rates, random rates, abrupt rates, or exponential rates; the fixed rate, i.e., the amount of data generated by the data source per second, is constant; random rate, i.e., the amount of data generated by the data source per second, is taken randomly within a range; the burst rate, i.e. the data source suddenly increases from a steady value to the limit rate in a short time; exponential rate, i.e. the amount of data generated by the data source per second increases according to an exponential function and cycles through the period;

the streaming scene and application constructor is responsible for constructing scenes and applications covering stream data calculation features, and the scenes and the applications are from two aspects, namely stream data processing scenes frequently encountered in actual life and test cases provided by a current stream processing framework test benchmark; meanwhile, the streaming scene and application constructor particularly considers a window mechanism in streaming processing, and performs parameter test of control variables by changing different window influence parameter values;

the performance data acquisition tool acquires performance indexes in the performance test process;

the performance data analysis tool processes and analyzes the performance indexes collected by the performance collection tool to reflect the influence of the parameter change on the performance indexes and diagnose the bottleneck of the frame for flow data processing;

the parameters comprise data parameters, application parameters or system parameters;

the system parameters comprise the memory, the CPU, the Slave number or the maximum parallelism of the cluster;

the performance index throughput rate, delay, back pressure, system resources, node processing rate, node data volume or node buffer pool usage;

the application is a window mode, the application parameter is a window parameter, and the window parameter includes a window type, parallelism, or window size.

2. A method for implementing a performance benchmarking system using the big data flow processing framework of claim 1, comprising the steps of:

(1) deploying a test cluster, and configuring system parameters of the framework;

(2) selecting a scene and an application, configuring data parameters, application parameters and test parameters, determining a plurality of values of a certain parameter to be tested, keeping other parameters unchanged, and running the application on the cluster to perform a plurality of groups of tests;

(3) running the application for testing, starting a performance acquisition tool at the same time, and waiting for the completion of the test;

(4) in the testing process, the performance acquisition tool periodically accesses the testing cluster to collect performance indexes, and persists the collected performance indexes to a storage device after the testing is finished;

(5) and when the application is finished or finished on the test cluster, analyzing the performance data acquired by the performance acquisition tool, comparing the influence of the change of the parameter to be tested on the performance index, analyzing layer by layer from top to bottom, and positioning the bottleneck point of the frame.

Images