CN107609026B - Data intensive application integration test method and system - Google Patents
Data intensive application integration test method and system Download PDFInfo
- Publication number
- CN107609026B CN107609026B CN201710674724.8A CN201710674724A CN107609026B CN 107609026 B CN107609026 B CN 107609026B CN 201710674724 A CN201710674724 A CN 201710674724A CN 107609026 B CN107609026 B CN 107609026B
- Authority
- CN
- China
- Prior art keywords
- data
- application
- intensive
- intensive application
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a data intensive application integration test method and a system, the method generates data intensive application through automatic customization and verifies whether the data intensive application meets the data intensive characteristics, and storing the data-intensive application after verifying that the data-intensive application satisfies the data-intensive characteristics, solves the technical problem that the prior art cannot automatically customize, generate and verify the data-intensive application, can not only automatically customize and generate the data-intensive application, but also can verify the automatically customized data-intensive application, and after verifying that the data-intensive application has data-intensive characteristics, the data-intensive application is stored, a stable and reliable data-intensive application test case is provided for researchers, and the process automation improves the generation efficiency of the data-intensive application.
Description
Technical Field
The invention relates to the technical field of information data processing, in particular to a data intensive application integration test method and system.
Background
With the development and popularization of internet technology, various internet technology-based applications are developed vigorously, and with the increase in the number of users of these applications and the exponential rise of user data, distributed, heterogeneous and data-intensive features become important features of these applications, which are referred to as data-intensive applications. The size of the data processed by such applications is also an important feature, and the size of the data set usually reaches TB or even PB. The data set is large in scale, and the complex dependency relationship among data is also one of the important characteristics of data intensive application.
In a traditional computing mode, although a supercomputer can solve the problem of computing mass data, the use range of the supercomputer is limited, so that more and more internet services gradually convert computing requirements into requirements for data processing and data into useful knowledge and information to realize data intelligence requirements when functions are realized. In traditional computing environments, small-scale data management problems are the focus of research, and with considerable advances and efforts, various data integration and query optimization methods have been successfully proposed and validated. However, the existing research results cannot automatically customize, generate and verify the data-intensive application when processing the data-intensive application, and aiming at the problem, the invention provides a test method and a test system for the data-intensive application.
Disclosure of Invention
The invention aims to provide a data intensive application integration test method and a data intensive application integration test system, which are used for solving the technical problem that the prior art cannot automatically customize, generate and verify data intensive applications.
In order to solve the technical problem, the data intensive application integration test method provided by the invention comprises the following steps:
automatically customizing and generating data intensive applications;
verifying whether the data-intensive application satisfies the data-intensive characteristics, and storing the data-intensive application after verifying that the data-intensive application satisfies the data-intensive characteristics.
Further, automatically customizing the type of data-intensive application generated includes:
MapReduce-type data-intensive applications and IOZones-type data-intensive applications, which in turn include DataReduction and WordCount-type data-intensive applications.
Further, automated customization of generation of data intensive applications of the IOZones type includes:
obtaining range information of the IOZones type data intensive application input by a user, wherein the range information comprises maximum and minimum node numbers, maximum and minimum widths, layer numbers and RW numbers;
initializing the number of nodes of the first layer and the last layer to be 1, and acquiring a random number between the maximum width and the minimum width of each of the other layers as the number of nodes of the layer;
randomly selecting a node in each layer, initializing the task of the node into RW, randomly distributing the rest RW tasks to the nodes which are not initialized yet, and initializing all the nodes which are not initialized as R tasks;
starting from the second layer, randomly selecting a RW of the previous layer as a parent node of the node at each layer to ensure that the degree of entry of each node is larger than 0, checking whether the degree of exit of each RW task node is larger than 0, if the degree of exit of a RW task node is equal to 0, selecting a node from the next layer of the layer where the node is located as a child node of the node to ensure that the degree of exit of the RW task node is larger than 0, and finally generating an iozones _ dag.txt result file;
reading the iteration times, the ranges of the read mode, the write mode and the read-write mode from the configuration file, randomly selecting a mode to initialize the command information of each node task, and storing the command information into an iozones _ command.
Further, verifying that the data-intensive application satisfies the data-intensive characteristics includes:
acquiring a directed acyclic graph of a data intensive application;
scheduling and executing the data intensive application according to the directed acyclic graph of the data intensive application, and acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application in the scheduling and executing process, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
and judging whether the average CPU utilization rate of the data-intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if so, judging that the data-intensive application meets the data-intensive characteristics, and otherwise, judging that the data-intensive application does not meet the data-intensive characteristics.
Further, verifying that the data-intensive application satisfies the data-intensive characteristics includes:
acquiring a directed acyclic graph of a data intensive application;
scheduling and executing the data-intensive application branches according to the directed acyclic graph of the data-intensive application, and calculating the average CPU utilization rate and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data-intensive application in the executing process, wherein the calculation formula for calculating the I/O communication calculation ratio of each application branch is as follows;
and judging whether the proportion of the application branches with the average CPU utilization rate of less than 50% or the I/O communication calculation ratio of more than 10 in the application branches reaches 70%, if so, judging that the data-intensive application meets the data-intensive characteristics, and otherwise, judging that the data-intensive application does not meet the data-intensive characteristics.
Further, verifying that the data-intensive application satisfies the data-intensive characteristics includes:
acquiring a directed acyclic graph of a data intensive application;
scheduling and executing the data intensive application according to the directed acyclic graph of the data intensive application, and acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application in the scheduling and executing process, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
judging whether the average CPU utilization rate of the data intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if not, judging that the data intensive application does not meet the data intensive characteristics, if so, scheduling and executing the data intensive application branches, and calculating the average CPU utilization rate and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data intensive application in the executing process, wherein the calculation formula for calculating the I/O communication calculation ratio of each application branch is as follows;
and judging whether the proportion of the application branches with the average CPU utilization rate of less than 50% or the I/O communication calculation ratio of more than 10 in the application branches reaches 70%, if so, judging that the data-intensive application meets the data-intensive characteristics, and otherwise, judging that the data-intensive application does not meet the data-intensive characteristics.
The invention provides a data intensive application integration test system, which comprises:
a data intensive application incremental library for automated custom generation of data intensive applications;
a data intensive application verification model for verifying whether the data intensive application satisfies the data intensive characteristics;
a data-intensive application memory to store the data-intensive application upon verifying that the data-intensive application satisfies the data-intensive characteristic.
Further, data intensive application incrementation libraries include:
IOZones customizer and MapReduce customizer, wherein the Mapreduce customizer in turn includes a DataReduction and WordCount customization module.
Further, the IOZones customizer includes:
the range information acquisition module is used for acquiring range information of the IOZones type data intensive application input by a user, wherein the range information comprises maximum and minimum node numbers, maximum and minimum widths, layer numbers and RW numbers;
the node number determining module is used for initializing the number of the nodes of the first layer and the last layer to be 1, and acquiring a random number between the maximum width and the minimum width of each of the other layers as the number of the nodes of the layer;
the task allocation module is used for randomly selecting a node in each layer, initializing the task of the node into RW, randomly allocating the rest RW tasks to the nodes which are not initialized yet, and initializing all the nodes which are not initialized as R tasks;
the association relation establishing module is used for starting from the second layer, randomly selecting a RW of the previous layer as a parent node of the node at each layer to ensure that the degree of entry of each node is larger than 0, checking whether the degree of exit of each RW task node is larger than 0, if the degree of exit of the RW task node is equal to 0, selecting a node from the next layer of the layer where the node is located as a child node of the node to ensure that the degree of exit of the RW task node is larger than 0, and finally generating an iozones _ dag.txt result file;
and the task initialization module is used for reading the iteration times, the ranges of the read mode, the write mode and the read-write mode from the configuration file, randomly selecting one mode to initialize the command information of each node task, and storing the command information into an iozones _ command.
Further, the data application verification model includes:
an overall verification model and/or a branch verification model, and the overall verification model includes:
the directed acyclic graph acquisition module is used for acquiring a directed acyclic graph of the data intensive application;
the integral I/O communication calculation ratio calculation module is used for scheduling and executing the data intensive application according to the directed acyclic graph of the data intensive application and acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application in the scheduling and executing process, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
the integral judgment module is used for judging whether the average CPU utilization rate of the data intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if so, judging that the data intensive application meets the data intensive characteristics, and otherwise, judging that the data intensive application does not meet the data intensive characteristics;
the branch verification model includes:
the directed acyclic graph acquisition module is used for acquiring a directed acyclic graph of the data intensive application;
the application branch I/O communication calculation ratio calculation module is used for scheduling and executing the data intensive application branches according to the directed acyclic graph of the data intensive application, and calculating the average CPU utilization ratio and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data intensive application in the execution process, wherein the calculation formula for calculating the I/O communication calculation ratio of each application branch is as follows;
and the branch judgment module is used for judging whether the application branch proportion with the average CPU utilization rate of less than 50% or the I/O communication calculation ratio of more than 10 reaches 70% in the application branches, if so, judging that the data intensive application meets the data intensive characteristics, and otherwise, judging that the data intensive application does not meet the data intensive characteristics.
Compared with the prior art, the invention has the advantages that:
according to the data intensive application integration test method and system, the data intensive application is generated through automatic customization, whether the data intensive application meets the data intensive characteristics or not is verified, and the data intensive application is stored after the data intensive application meets the data intensive characteristics is verified, so that the technical problem that the data intensive application cannot be generated and verified through automatic customization in the prior art is solved, the data intensive application can be generated through automatic customization, the data intensive application generated through automatic customization can be verified, the data intensive application generated through automatic customization is stored after the data intensive application is verified to have the data intensive characteristics, and a reliable and stable test case of the data intensive application is further provided for a user.
In addition to the objects, features and advantages described above, other objects, features and advantages of the present invention are also provided. The present invention will be described in further detail below with reference to the drawings.
Drawings
FIG. 1 is a flow chart of a data intensive application integration testing method of a preferred embodiment of the present invention;
FIG. 2 is a partition-map branching structure of a DAG map for a MapReduce-type data intensive application of the preferred embodiment of the present invention;
FIG. 3 is a map-reduce branching structure of a DAG map for a MapReduce-type data intensive application of the preferred embodiment of the present invention;
FIG. 4 is a reduce-reduce branching structure of a DAG map for a MapReduce-type data intensive application of the preferred embodiment of the present invention;
FIG. 5 is a flow diagram of the automated customization of the present invention to generate data intensive applications of the IOZones type;
FIG. 6 is a flow chart of a preferred embodiment of the present invention for verifying whether a data-intensive application satisfies a data-intensive characteristic;
FIG. 7 is a block diagram of the architecture of a data intensive application integration test system in accordance with a preferred embodiment of the present invention;
FIG. 8 is a block diagram of the IOZones customizer of the preferred embodiment of the present invention;
FIG. 9 is a block diagram of the structure of a data application verification model in accordance with a preferred embodiment of the present invention.
Illustration of the drawings:
10. incremental library of data intensive applications; 20. a data intensive application verification model; 30. a data intensive application memory; 40. an IOZones customizer; 50. a MapReduce customizer; 60. a DataReduction customization module; 70. a WordCount customization module; 80. a range information acquisition module; 90. a node number determination module; 100. a task allocation module; 110. an association relationship establishing module; 120. a task initialization module; 130. integrally verifying the model; 140. a branch verification model; 150. a directed acyclic graph acquisition module; 160. the integral I/O communication calculation ratio calculation module; 170. an integral judgment module; 180. calculating the ratio by using a branch I/O communication calculation module; 190. and a branch judgment module.
Detailed Description
In order to facilitate an understanding of the invention, the invention will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of the invention is not limited to the specific embodiments below.
Referring to fig. 1, a preferred embodiment of the present invention provides a data intensive application integration test method, including:
step S101, automatically customizing and generating data intensive application;
step S102, verifying whether the data-intensive application meets the data-intensive characteristics, and storing the data-intensive application after verifying that the data-intensive application meets the data-intensive characteristics.
According to the data intensive application integration test method, the data intensive application is generated through automatic customization, whether the data intensive application meets the data intensive characteristics or not is verified, the data intensive application is stored after the data intensive application meets the data intensive characteristics is verified, the technical problem that the data intensive application cannot be generated and verified through automatic customization in the prior art is solved, the data intensive application can be generated through automatic customization, the data intensive application generated through automatic customization can be verified, the data intensive application generated through automatic customization is stored after the data intensive application is verified to have the data intensive characteristics, a stable and reliable data intensive application test case is provided for researchers, and the process automation is more capable of improving the generation efficiency of the data intensive application.
Specifically, the automatic customized generation of the data-intensive applications in the embodiment is mainly to automatically generate corresponding data-intensive applications according to application information input by a user, and store the data-intensive applications locally in the form of a DAG graph for later use. And the accuracy and the reliability of the data intensive application generated by automatic customization are improved by verifying the data intensive application generated by automatic customization.
In a big Data environment, Data Intensive Applications (DIA) centering on Data processing are increasing. Data-intensive applications are dominated by I/O operations, whereas for compute-intensive applications, the CPU utilization is much higher than that of data-intensive applications during application execution, since the computation tasks are dominated. For data intensive applications, CPU utilization is not high because the CPU has a lot of time waiting for I/O operations. The data-intensive applications are mainly characterized in that:
(1) massive independent data processing tasks can be distributed on different nodes of different computer distributed systems to be executed;
(2) high-density mass data I/O throughput requirements;
(3) most applied processes should be dominated by the data stream.
In addition, the data-intensive application (DIA) generated by the embodiment of the present invention is mainly stored locally in two files, one file stores the dependency relationship of the DAG graph, and the other file stores the execution command information of each node in the DIA. The user can schedule and execute the application by using the two files and combining various scheduling algorithms, so that the user can not only study the static characteristics of the DIA, but also study the dynamic characteristics of the DIA. The efficiency of the scheduling algorithm may also be studied using DIA, etc.
Optionally, automatically customizing the type of data-intensive application comprises:
MapReduce-type data-intensive applications and IOZones-type data-intensive applications, which in turn include DataReduction and WordCount-type data-intensive applications.
Specifically, the types of the data-intensive applications generated by the automatic customization in the present embodiment include MapReduce type data-intensive applications and IOZones type data-intensive applications, and the MapReduce type data-intensive applications include DataReduction and WordCount type data-intensive applications. And for WordCount and DataDedupling type data intensive applications, the specific implementation of Map and Reduce applications is different.
Data intensive applications based on MapReduce type require that the application be converted into a DAG graph before subsequent validation and scheduling work can be performed. The branching structure and specific atomic operations of the data intensive application DAG graph are given below, both types WordCount and DataDedu functionality.
The DAG graph for MapReduce-type data-intensive applications mainly includes three basic branch structures, namely, a partition-map branch structure, a map-reduce branch structure, and a reduce-reduce branch structure, which are shown in fig. 2, 3, and 4, respectively.
It should be noted that, in graph theory, if a directed graph cannot go from any vertex back to the point through several edges, the graph is a directed acyclic graph (DAG graph). A directed acyclic graph may not be convertible to a tree because a directed graph does not necessarily form a ring from one point to another point through two routes, but any directed tree is a directed acyclic graph. DAGs can be used to model some different kinds of structures in mathematics and computer science. Due to constraints that some tasks must execute earlier than others, the set of tasks that must be ordered into a queue can be presented by a DAG graph, where each vertex represents a task and each edge represents a constraint, and topological ordering algorithms can be used to generate a valid sequence. DAGs may also be used to model the process of information passing through a network of processors in a consistent direction. Reachability relationships within the DAG constitute a local order, and any limited local order may be presented by the DAG using reachability. Furthermore, DAGs are representative of overlapping subsequences of a sequence set that can be space efficient.
In addition, the operation of a specific task for a MapReduce type of data intensive application is (taking WordCount as an example):
partition operation: part is { indata, map _ count }, indata is input data, that is, txt text, and map _ count is the number of map tasks. After Partition is finished, input data are written into text files with the same number as the map tasks, namely the input files are used as input files of the map _1, the map _2, the …, the map _ n and other tasks.
And Map operation: map is { index, reduce _1, reduce _2, …, reduce _ n }, index is input data, and reduce _1, reduce _2, …, reduce _ n is the number of the reduce task written by the Map task. The Map operation sequentially reads the character strings in the input file and outputs a key value pair, wherein key is the value of the character strings, and value is 1. And additionally writing the key value pair into the corresponding output file reduce _ n.txt without accumulation.
Reduce operation: reduce is { indata, result }, indata is input data, and result is an output file of the Reduce task. And reading the key value pair in the corresponding input file Reduce _ n.txt by Reduce operation, and merging the key values which are the same. And finally writing the merging result into the corresponding file result _ n.txt.
The embodiment of the invention provides a new data intensive application IOZones DIA, and integrates two typical data intensive applications based on a MapReduce framework, namely WordCount and DataReduction. A plurality of data-intensive application test cases are provided for selection by a user, so that the selection of the data-intensive applications of the test system is diversified.
Referring to fig. 5, the present embodiment automatically customizes the generation of data-intensive applications of the IOZones type, including:
step S501, obtaining range information of the IOZones type data intensive application input by a user, wherein the range information comprises maximum and minimum node numbers, maximum and minimum widths, layer numbers and RW numbers;
step S502, initializing the number of nodes of the first layer and the last layer to be 1, and taking a random number between the maximum width and the minimum width of each of the other layers as the number of nodes of the layer;
step S503, randomly selecting a node in each layer and initializing the task of the node to be RW, simultaneously randomly distributing the rest RW tasks to the nodes which are not initialized yet, and initializing all the nodes which are not initialized to be R tasks;
step S504, starting from the second layer, the node of each layer randomly selects a RW of the previous layer as the parent node thereof to ensure that the degree of entry of each node is necessarily greater than 0, checks whether the degree of exit of each RW task node is greater than 0, if the degree of exit of a RW task node is equal to 0, selects a node from the next layer of the layer number where the node is located as the child node of the node to ensure that the degree of exit of the RW task node is necessarily greater than 0, and finally generates an iozones _ dag.txt result file;
step S505, reading the iteration number, the range of the read mode, the write mode, and the read-write mode from the configuration file, randomly selecting a mode to initialize the command information of each node task, and storing the command information in an iozones _ command.
The IOzone is a file system test benchmark tool. The read-write performance of the file system in different operating systems can be tested. The performance of the hard disk can be determined in different modes such as write, re-write, read, re-read, random read, random write, random mmix, backup read, record read, stridded read, fwrite, frewrite, fread, freread, mmap, async I/0 and the like. The iozone task is mainly based on io operations, and therefore, the iozone is a genuine data-intensive task. However, each iozone operation is an independent test task, and the iozone operations are atomic, so that the iozone operations cannot be combined into one data-intensive task.
To address this problem, the present embodiment provides a new typical data-intensive application, IOZones, and automatically customizes the generation of the data-intensive application based on the three core atomic operations (R, W, RW) in iozone. The Read operation comprises 6 modes, namely Randomread, Read, Reread, Backwardread, Stridedread and aio-Read. The Write operation contains 5 modes, Write, RandomWrite, Rewrite, Stridedwrite and aio-Write, respectively. The readandwrite operation is a combination of different modes of the read operation and the write operation.
The specific program algorithm for the automated customized generation of data intensive applications of the IOZones type of the present embodiment is as follows:
in the embodiment, the data intensive application of the IOZones type is generated through automatic customization, a new typical data intensive application IOZones is provided, and then the test case of the data intensive application of the IOZones type is provided for the user.
The customized algorithm of the IOZones DIA provided by the embodiment of the invention automatically generates the corresponding IOZones DIA according to the rough information of the DIA input by a user. The algorithm may implement IOZones DIA automated generation. After the generation is successful, two files are generated, namely a DAG description file (IOZones _ dag.txt) of the IOZones DIA, and an executable command information file (IOZones _ command.txt) corresponding to each node of the IOZones DIA, and the IOZones DIA can be scheduled to execute by using the files.
The iozones _ dag.txt in this embodiment is specifically as follows:
txt describes key information and association relationship information between nodes in a DAG graph, the DAG graph contains 10 nodes and 9 edges, and "links 10135" means that the first edge connects node 1 and node 0, and the weight is 35.
And the details of the command information file iozones _ command.txt used for saving each node are as follows:
the file describes the task information for each node. "/home/peter/DIA/src/tasks/readandWrite.o 1/home/peter/DIA/src/tasks/data/c0.txt/home/peter/DIA/src/data/task/data/c 2.txt FREEDFREE 35002", 2 refers to the 2 nd task, which is a RW task,/home/peter/DIA/src/tasks/readandWrite.o refers to path information of an executing task, and the others are parameter information of the executing file, FREAD is a read mode, Free is a write mode, and 3500 is the number of iterations of writing the file (3500 is represented by 3500 characters written into the file).
Optionally, the verifying whether the data-intensive application satisfies the data-intensive characteristic according to the embodiment includes:
acquiring a directed acyclic graph of a data intensive application;
scheduling and executing the data intensive application according to the directed acyclic graph of the data intensive application, and acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application in the scheduling and executing process, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
and judging whether the average CPU utilization rate of the data-intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if so, judging that the data-intensive application meets the data-intensive characteristics, and otherwise, judging that the data-intensive application does not meet the data-intensive characteristics.
Since the data-intensive application obtained by the embodiment is generated by automatic customization, in order to more accurately obtain the data-intensive application with the data-intensive application characteristics, the embodiment performs data-intensive characteristic verification on the data-intensive application generated by automatic customization.
Specifically, the DIA (data intensive application) is scheduled to execute according to a generated DAG graph (directed acyclic graph) of the DIA, and an average CPU utilization (CPU) of the system, a computation time (computetime) of the DIA, an I/O time (epoch) of the DIA, and a transmission time (transmissiontime) of the DIA are acquired during the execution of the DIA. And calculating the I/O Communication Calculation Ratio (CCR) according to a formula. When the overall average CPU utilization is less than 50% or the I/O communication computation ratio is greater than 10, then the DIA is determined to satisfy the specific characteristics of the data-intensive application, otherwise the data-intensive application is determined not to satisfy the data-intensive characteristics.
In the embodiment, whether the data-intensive application meets the data-intensive characteristics is verified by calculating the average CPU utilization ratio and the I/O communication calculation ratio of the data-intensive application, so that the data-intensive application with the data-intensive application characteristics can be obtained more accurately.
Optionally, the verifying whether the data-intensive application satisfies the data-intensive characteristic according to the embodiment includes:
acquiring a directed acyclic graph of a data intensive application;
scheduling and executing the data-intensive application branches according to the directed acyclic graph of the data-intensive application, and calculating the average CPU utilization rate and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data-intensive application in the executing process, wherein the calculation formula for calculating the I/O communication calculation ratio of each application branch is as follows;
and judging whether the proportion of the application branches with the average CPU utilization rate of less than 50% or the I/O communication calculation ratio of more than 10 in the application branches reaches 70%, if so, judging that the data-intensive application meets the data-intensive characteristics, and otherwise, judging that the data-intensive application does not meet the data-intensive characteristics.
In the embodiment, whether the data-intensive application meets the data-intensive characteristics is verified by calculating the average CPU utilization and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data-intensive application, so that the data-intensive application with the data-intensive application characteristics can be more accurately obtained.
Referring to fig. 6, optionally, the present embodiment verifying whether the data-intensive application satisfies the data-intensive characteristics includes:
step S601, acquiring a directed acyclic graph of the data intensive application;
step S602, according to the directed acyclic graph of the data intensive application, scheduling and executing the data intensive application, and in the process of scheduling and executing, acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
step S603, judging whether the average CPU utilization rate of the data intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if not, judging that the data intensive application does not meet the data intensive characteristics, if so, scheduling and executing the data intensive application branches, and calculating the average CPU utilization rate and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data intensive application in the executing process, wherein the calculation formula for calculating the I/O communication calculation ratio of each application branch is as follows;
step S604, judging whether the application branch proportion of which the average CPU utilization rate is less than 50% or the I/O communication calculation ratio is more than 10 in the application branch reaches 70%, if so, judging that the data intensive application meets the data intensive characteristics, otherwise, judging that the data intensive application does not meet the data intensive characteristics.
In order to more accurately verify whether the data-intensive application generated by the automatic customization meets the data-intensive characteristics, the embodiment further performs branch verification on the data-intensive application after determining that the average CPU utilization of the data-intensive application is less than 50% or the I/O communication calculation ratio is greater than 10. Specifically, each edge in the DAG is tested separately when DIA is executed, and the average CPU utilization rate of the ith edge and the I/O time of the ith edge are obtained in the process of executing each edge in the DAG graphiComputing time of ith edgeiAnd transmission time of ith edgeiAnd calculating the I/O communication calculation ratio CCR of the ith edge through a formulai. When the average CPU utilization of the ith edge is less than 50% or the I/O communication calculation ratio is greater than 10, the edge passes the verification. And when 70% of edges pass the verification, judging that the data-intensive application meets the data-intensive characteristics and belongs to the data-intensive application, otherwise, judging that the data-intensive application does not meet the data-intensive characteristics and does not belong to the data-intensive application.
According to the embodiment, the data intensive application which is generated by automatic customization can be verified doubly, so that the data intensive application which accords with the data intensive characteristics can be obtained more accurately, and a test case of the more accurate data intensive application can be obtained.
Referring to fig. 7, the data intensive application integration test system of the present invention includes:
a data intensive application incremental library 10 for automated custom generation of data intensive applications;
a data intensive application verification model 20 for verifying whether the data intensive application satisfies the data intensive characteristics;
a data-intensive application store 30 for storing the data-intensive application upon verifying that the data-intensive application satisfies the data-intensive characteristics.
The data intensive application integration test system of the embodiment of the invention generates the data intensive application through automatic customization, verifies whether the data intensive application meets the data intensive characteristics, and stores the data intensive application after verifying that the data intensive application meets the data intensive characteristics, thereby solving the technical problem that the data intensive application cannot be automatically customized and generated and verified in the prior art, not only being capable of automatically customizing and generating the data intensive application, but also being capable of verifying the data intensive application automatically customized and generated, and storing the data intensive application after verifying that the data intensive application has the data intensive characteristics, thereby providing a reliable and stable test case of the data intensive application for a user.
FIG. 7 is a block diagram of the data intensive application integration test system of this embodiment, which contains three modules, DIA incremental library, DIA verification model and DIA memory.
The DIAL incremental library (DIA is a data intensive application) provides a series of interfaces for users to customize corresponding data intensive tasks, and two customizers, namely an IOZones customizer and a MapReduce customizer, are included in the DIAL incremental library. The two customizers automatically generate corresponding data-intensive applications from the user-entered application information and save the applications locally in the form of a DAG graph for later use.
The DIA verification model mainly verifies applications generated in the DIA incremental library. It is determined whether the newly generated application conforms to the characteristics of the data-intensive application. The DIA authentication model includes a set of authentication decision methods for data intensive applications. The user-customized DIA and the DIA that is system-owned are stored in the DIA storage. The user may also view the main information of the DIA in the DIA storage, including the number of nodes, the number of edges, the storage path of the DIA, and the nodes containing the maximum in-degree out-degree sum in the whole DAG graph.
Optionally, the data intensive application incrementation library 10 includes:
an IOZones customizer 40 and a MapReduce customizer 50, wherein the MapReduce customizer 50 in turn comprises a DataReduction customizations module 60 and a WordCount customizations module 70.
Referring to fig. 8, the IOZones customizer 40 of the present embodiment includes:
a range information obtaining module 80, configured to obtain range information of the data-intensive application of the IOZones type input by the user, where the range information includes maximum and minimum node numbers, maximum and minimum widths, number of layers, and number of RWs;
a node number determining module 90, configured to initialize the number of nodes in the first layer and the last layer to be 1, and obtain a random number between the maximum width and the minimum width of each of the remaining layers as the number of nodes in the layer;
a task allocation module 100, configured to randomly select a node in each layer and initialize the task of the node to be RW, and simultaneously randomly allocate the remaining RW tasks to nodes that have not been initialized yet, and initialize all the nodes that have not been initialized to be R tasks;
an association relationship establishing module 110, configured to start from the second layer, select a RW in the previous layer at random as its parent node to ensure that the degree of entry of each node is necessarily greater than 0, check whether the degree of exit of each RW task node is greater than 0, if there is a RW task node whose degree of exit is equal to 0, select a node from the next layer of the layer where the node is located as a child node of the node to ensure that the degree of exit of the RW task node is necessarily greater than 0, and finally generate an iozones _ dag.txt result file;
and the task initialization module 120 is configured to read the iteration number, the range of the read mode, the write mode, and the read-write mode from the configuration file, randomly select a mode to initialize the command information of each node task, and store the command information in an iozones _ command.
Referring to FIG. 9, the data intensive application verification model 20 includes:
a directed acyclic graph obtaining module 150, configured to obtain a directed acyclic graph for a data intensive application;
the overall I/O communication calculation ratio calculation module 160 is configured to schedule and execute the data-intensive application according to the directed acyclic graph of the data-intensive application, and obtain an average CPU utilization ratio and an I/O communication calculation ratio of the data-intensive application during the scheduling and executing process, where a calculation formula for calculating the I/O communication calculation ratio is as follows;
the integral judgment module 170 is configured to judge whether the average CPU utilization of the data-intensive application is less than 50% or whether the I/O communication computation ratio is greater than 10, if yes, it is judged that the data-intensive application satisfies the data-intensive characteristics, and if not, it is judged that the data-intensive application does not satisfy the data-intensive characteristics;
the branch verification model 140 includes:
a directed acyclic graph obtaining module 150, configured to obtain a directed acyclic graph for a data intensive application;
an application branch I/O communication calculation ratio calculation module 180, configured to schedule and execute the data-intensive application branches according to the directed acyclic graph of the data-intensive application, and calculate an average CPU utilization ratio and an I/O communication calculation ratio of each application branch of the directed acyclic graph of the data-intensive application in the execution process, where a calculation formula for calculating the I/O communication calculation ratio of each application branch is;
the branch determination module 190 is configured to determine whether the application branch ratio in the application branch, where the average CPU utilization is less than 50% or the I/O communication computation ratio is greater than 10, reaches 70%, if so, determine that the data-intensive application satisfies the data-intensive characteristic, otherwise, determine that the data-intensive application does not satisfy the data-intensive characteristic.
The DIA verification model of the embodiments of the present invention includes a set of verification decision methods for data intensive applications. The application to be verified is tested in the verification model in two aspects, on one hand, the overall verification test is carried out, the application is scheduled, the average CPU utilization rate, the calculation time, the I/O time and the transmission time during application execution are obtained, and whether the application belongs to the data intensive application or not is judged by calculating the I/O communication calculation ratio and the average CPU utilization rate during application execution. The other aspect is a branch validation test, which tests each branch in the DAG graph describing the application individually, obtains the average CPU utilization and I/O traffic computation ratio of each branch when executing, and passes the branch validation test when 70% of the number of branches passes validation. If both tests pass, the application is determined to be a data intensive application.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A data intensive application integration test method is characterized by comprising the following steps:
automatically custom generating a data-intensive application, wherein automatically custom generating a type of the data-intensive application comprises:
MapReduce type of data intensive applications, which in turn include DataReduction and WordCount type of data intensive applications, and IOZones type of data intensive applications, and automated custom generation of IOZones type of data intensive applications includes:
obtaining range information of data intensive application of an IOZones type input by a user, wherein the range information comprises maximum and minimum node numbers, maximum and minimum widths, layer numbers and RW numbers;
initializing the number of nodes of the first layer and the last layer to be 1, and acquiring a random number between the maximum width and the minimum width of each of the other layers as the number of nodes of the layer;
randomly selecting a node in each layer, initializing the task of the node into RW, randomly distributing the rest RW tasks to the nodes which are not initialized yet, and initializing all the nodes which are not initialized as R tasks;
starting from the second layer, randomly selecting a RW of the previous layer as a parent node of the node at each layer to ensure that the degree of entry of each node is larger than 0, checking whether the degree of exit of each RW task node is larger than 0, if the degree of exit of a RW task node is equal to 0, selecting a node from the next layer of the layer where the node is located as a child node of the node to ensure that the degree of exit of the RW task node is larger than 0, and finally generating an iozones _ dag.txt result file;
reading iteration times, a range of a read mode, a range of a write mode and a range of a read-write mode from a configuration file, randomly selecting a mode to initialize command information of each node task, and storing the command information into an iozones _ command.txt file;
verifying whether the data-intensive application satisfies the data-intensive characteristics, and storing the data-intensive application after verifying that the data-intensive application satisfies the data-intensive characteristics.
2. The data-intensive application integration test method of claim 1, wherein verifying whether the data-intensive application satisfies a data-intensive characteristic comprises:
obtaining a directed acyclic graph of the data intensive application;
scheduling and executing the data intensive application according to the directed acyclic graph of the data intensive application, and acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application in the scheduling and executing process, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
and judging whether the average CPU utilization rate of the data intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if so, judging that the data intensive application meets the data intensive characteristics, and otherwise, judging that the data intensive application does not meet the data intensive characteristics.
3. The data-intensive application integration test method of claim 1, wherein verifying whether the data-intensive application satisfies a data-intensive characteristic comprises:
obtaining a directed acyclic graph of the data intensive application;
scheduling and executing the data-intensive application branches according to the directed acyclic graph of the data-intensive application, and calculating the average CPU utilization rate and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data-intensive application in the executing process, wherein the calculation formula for calculating the I/O communication calculation ratio of each application branch is as follows;
and judging whether the proportion of the application branches with the average CPU utilization rate of less than 50% or the I/O communication calculation ratio of more than 10 in the application branches reaches 70%, if so, judging that the data intensive application meets the data intensive characteristics, and otherwise, judging that the data intensive application does not meet the data intensive characteristics.
4. The data-intensive application integration test method of claim 1, wherein verifying whether the data-intensive application satisfies a data-intensive characteristic comprises:
obtaining a directed acyclic graph of the data intensive application;
scheduling and executing the data intensive application according to the directed acyclic graph of the data intensive application, and acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application in the scheduling and executing process, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
judging whether the average CPU utilization rate of the data intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if not, judging that the data intensive application does not meet the data intensive characteristics, if so, scheduling and executing the data intensive application branches, and calculating the average CPU utilization rate and the I/O communication calculation ratio of each application branch of the directed acyclic graph of the data intensive application in the executing process, wherein the calculation formula for calculating the I/O communication calculation ratio of each application branch is as follows;
and judging whether the proportion of the application branches with the average CPU utilization rate of less than 50% or the I/O communication calculation ratio of more than 10 in the application branches reaches 70%, if so, judging that the data intensive application meets the data intensive characteristics, and otherwise, judging that the data intensive application does not meet the data intensive characteristics.
5. A data intensive application integration test system, the system comprising:
a data-intensive application incrementation library for automated, custom generation of data-intensive applications, wherein the data-intensive application incrementation library comprises:
an IOZones customizer and a MapReduce customizer, wherein the Mapreduce customizer in turn comprises a DataReduction and WordCount customization module, and the IOZones customizer comprises:
the range information acquisition module is used for acquiring range information of the IOZones type data intensive application input by a user, wherein the range information comprises maximum and minimum node numbers, maximum and minimum widths, layer numbers and RW numbers;
the node number determining module is used for initializing the number of the nodes of the first layer and the last layer to be 1, and acquiring a random number between the maximum width and the minimum width of each of the other layers as the number of the nodes of the layer;
the task allocation module is used for randomly selecting a node in each layer, initializing the task of the node into RW, randomly allocating the rest RW tasks to the nodes which are not initialized yet, and initializing all the nodes which are not initialized as R tasks;
the association relation establishing module is used for starting from the second layer, randomly selecting a RW of the previous layer as a parent node of the node at each layer to ensure that the degree of entry of each node is larger than 0, checking whether the degree of exit of each RW task node is larger than 0, if the degree of exit of the RW task node is equal to 0, selecting a node from the next layer of the layer where the node is located as a child node of the node to ensure that the degree of exit of the RW task node is larger than 0, and finally generating an iozones _ dag.txt result file;
the task initialization module is used for reading the iteration times, the ranges of a read mode, a write mode and a read-write mode from the configuration file, randomly selecting one mode to initialize the command information of each node task, and storing the command information into an iozones _ command.txt file;
a data intensive application verification model for verifying whether the data intensive application satisfies a data intensive characteristic;
a data-intensive application store to store the data-intensive application upon verification that the data-intensive application satisfies a data-intensive characteristic.
6. The data intensive application integration test system of claim 5, wherein the data application verification model comprises:
an overall verification model and/or a branching verification model, and the overall verification model comprises:
the directed acyclic graph acquisition module is used for acquiring a directed acyclic graph of the data intensive application;
the integral I/O communication calculation ratio calculation module is used for scheduling and executing the data intensive application according to the directed acyclic graph of the data intensive application and acquiring the average CPU utilization rate and the I/O communication calculation ratio of the data intensive application in the scheduling and executing process, wherein the calculation formula for calculating the I/O communication calculation ratio is as follows;
the integral judgment module is used for judging whether the average CPU utilization rate of the data intensive application is less than 50% or whether the I/O communication calculation ratio is greater than 10, if so, judging that the data intensive application meets the data intensive characteristics, and otherwise, judging that the data intensive application does not meet the data intensive characteristics;
the branch verification model includes:
the directed acyclic graph acquisition module is used for acquiring a directed acyclic graph of the data intensive application;
an application branch I/O communication calculation ratio calculation module, configured to schedule and execute the data-intensive application branches according to the directed acyclic graph of the data-intensive application, and calculate an average CPU utilization ratio and an I/O communication calculation ratio of each application branch of the directed acyclic graph of the data-intensive application in an execution process, where a calculation formula for calculating the I/O communication calculation ratio of each application branch is;
and the branch judgment module is used for judging whether the application branch proportion with the average CPU utilization rate of less than 50% or the I/O communication calculation ratio of more than 10 in the application branches reaches 70%, if so, judging that the data intensive application meets the data intensive characteristics, and otherwise, judging that the data intensive application does not meet the data intensive characteristics.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710674724.8A CN107609026B (en) | 2017-08-09 | 2017-08-09 | Data intensive application integration test method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710674724.8A CN107609026B (en) | 2017-08-09 | 2017-08-09 | Data intensive application integration test method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107609026A CN107609026A (en) | 2018-01-19 |
| CN107609026B true CN107609026B (en) | 2020-11-06 |
Family
ID=61064915
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710674724.8A Active CN107609026B (en) | 2017-08-09 | 2017-08-09 | Data intensive application integration test method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107609026B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101196817A (en) * | 2008-01-04 | 2008-06-11 | 福建星网锐捷网络有限公司 | Test case generating method and system |
| CN103593452A (en) * | 2013-11-21 | 2014-02-19 | 北京科技大学 | Data intensive computing cost optimization method based on MapReduce mechanism |
| CN104077218A (en) * | 2013-03-29 | 2014-10-01 | 百度在线网络技术（北京）有限公司 | Test method and device of MapReduce distributed system |
| CN104298604A (en) * | 2014-11-06 | 2015-01-21 | 哈尔滨工业大学 | Testing system and testing method for robustness of cloud service |
| CN105302717A (en) * | 2015-09-30 | 2016-02-03 | 浪潮集团有限公司 | Detection method and apparatus for big data platform |
| CN105589803A (en) * | 2014-10-24 | 2016-05-18 | 阿里巴巴集团控股有限公司 | Method and terminal device for generating test tool |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9146830B2 (en) * | 2012-10-26 | 2015-09-29 | Jsmapreduce Corporation | Hybrid local/remote infrastructure for data processing with lightweight setup, powerful debuggability, controllability, integration, and productivity features |
-
2017
- 2017-08-09 CN CN201710674724.8A patent/CN107609026B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101196817A (en) * | 2008-01-04 | 2008-06-11 | 福建星网锐捷网络有限公司 | Test case generating method and system |
| CN104077218A (en) * | 2013-03-29 | 2014-10-01 | 百度在线网络技术（北京）有限公司 | Test method and device of MapReduce distributed system |
| CN103593452A (en) * | 2013-11-21 | 2014-02-19 | 北京科技大学 | Data intensive computing cost optimization method based on MapReduce mechanism |
| CN105589803A (en) * | 2014-10-24 | 2016-05-18 | 阿里巴巴集团控股有限公司 | Method and terminal device for generating test tool |
| CN104298604A (en) * | 2014-11-06 | 2015-01-21 | 哈尔滨工业大学 | Testing system and testing method for robustness of cloud service |
| CN105302717A (en) * | 2015-09-30 | 2016-02-03 | 浪潮集团有限公司 | Detection method and apparatus for big data platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107609026A (en) | 2018-01-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108595157B (en) | Block chain data processing method, device, equipment and storage medium | |
| US8195648B2 (en) | Partitioned query execution in event processing systems | |
| US8578348B2 (en) | System and method of cost oriented software profiling | |
| Liu et al. | Multi-objective scheduling of scientific workflows in multisite clouds | |
| CN100426239C (en) | Dependency graph parameter scoping | |
| US9454384B2 (en) | Custom operating system via a web-service | |
| US20170235661A1 (en) | Integration of Software Systems via Incremental Verification | |
| CN113095474A (en) | Resource usage prediction for deep learning models | |
| Wozniak et al. | An optimization approach for the synthesis of autosar architectures | |
| Van Hee et al. | Soundness of resource-constrained workflow nets | |
| Hudson et al. | libEnsemble users manual | |
| US8086455B2 (en) | Model development authoring, generation and execution based on data and processor dependencies | |
| US8423980B1 (en) | Methods for handling inlined functions using sample profiles | |
| US10268461B2 (en) | Global data flow optimization for machine learning programs | |
| Ansel | Autotuning programs with algorithmic choice | |
| US20120303560A1 (en) | System and method for development of a system architecture | |
| US11262986B2 (en) | Automatic software generation for computer systems | |
| CN107609026B (en) | Data intensive application integration test method and system | |
| US9495141B1 (en) | Expanding inline function calls in nested inlining scenarios | |
| Acevedo et al. | A Critical Path File Location (CPFL) algorithm for data-aware multiworkflow scheduling on HPC clusters | |
| CN112800425A (en) | Code analysis method and device based on graph calculation | |
| Wang et al. | Monte carlo tree search for task mapping onto heterogeneous platforms | |
| Micconi et al. | Multi-ASIP platform synthesis for real-time applications | |
| Dietze et al. | Search-Based Scheduling for Parallel Tasks on Heterogeneous Platforms | |
| Metzler et al. | Efficient verification of program fragments: Eager POR |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |