CN105302730A - Calculation model detection method, testing server and service platform - Google Patents
Calculation model detection method, testing server and service platform Download PDFInfo
- Publication number
- CN105302730A CN105302730A CN201510903291.XA CN201510903291A CN105302730A CN 105302730 A CN105302730 A CN 105302730A CN 201510903291 A CN201510903291 A CN 201510903291A CN 105302730 A CN105302730 A CN 105302730A
- Authority
- CN
- China
- Prior art keywords
- computation model
- storage server
- instruction
- query statement
- sql query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention provides a calculation model detection method, a testing server and a service platform. The method includes the steps that a target calculation model is acquired and deployed on at least one storage server; a TPC-DS testing assembly is acquired, a first detection instruction is generated and sent to the target calculation model, and the first detection instruction instructs the storage servers corresponding to the target calculation model to perform corresponding service processing on structured data; first resource consumption information is acquired and stored; a bigbench testing assembly is acquired, a second detection instruction is generated and sent to the target calculation model, and the second detection instruction instructs the storage servers corresponding to the target calculation model to perform corresponding service processing on unstructured data; second resource consumption information is acquired and stored. According to the technical scheme, the accuracy of testing results can be improved.
Description
Technical field
The present invention relates to field of computer technology, particularly a kind of method of detection computations model, testing server and business platform.
Background technology
Along with the commercialization of large data framework technology such as increase income Hapdoop, Map/Reduce, Spark, HDFS, HBASE etc., computation model (i.e. data management system) application based on above-mentioned each large data framework is also more and more extensive.
Computation model can be used for large data management business, and the computation model based on the large data framework of difference possesses different data-handling efficiencies; At present, usual use TPC-DS test benchmark respectively to based on the large data framework of difference and the computation model realizing same target business detect, to obtain each computation model based on large data framework resource consumption information corresponding respectively, and then make user suitable computation model can be selected to realize corresponding large data management business in conjunction with practical business demand according to the resource consumption information got; But, needed for computation model, data to be processed generally include structural data and unstructured data, when processing accordingly respectively with unstructured data for structural data, the required resource taken is not identical, and TPC-DS test benchmark can only detect the data processing business of structure based data, therefore, the testing result accuracy for computation model is lower.
Summary of the invention
The invention provides a kind of method of detection computations model, testing server and business platform, the accuracy of testing result can be improved.
First aspect, the invention provides a kind of method of detection computations model, comprising:
S0: obtain target computation model, and target computation model is deployed at least one storage server;
S1: obtain TPC-DS test suite, produce first according to described TPC-DS test suite and detect instruction, send first to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for structural data;
S2: obtain the first resource consumption information that described at least one storage server corresponding described first detects instruction, and store described first resource consumption information;
S3: obtain bigbench test suite, produce second according to described bigbench test suite and detect instruction, send second to described target computation model and detect instruction, described second detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for unstructured data;
S4: obtain the Secondary resource consumption information that described at least one storage server corresponding described second detects instruction, and store described Secondary resource consumption information.
Further,
Described first detects instruction carries at least one first Structured Query Language (SQL) SQL query statement, also comprises:
When described target computation model fails to provide first object data subset according to each first SQL query statement respectively, the first SQL query statement that record is corresponding and described target computation model fail to provide according to the first SQL query statement the number of times of first object data subset;
And,
Described second detects instruction carries at least one second SQL query statement, also comprises:
When described target computation model fails to provide the second target data subset according to each second SQL query statement respectively, the second SQL query statement that record is corresponding and described target computation model fail to provide according to the second SQL query statement the number of times of the second target data subset.
Further,
Described at least one the storage server corresponding described first of described acquisition detects the first resource consumption information of instruction, comprise: obtain each for carrying out the storage server of corresponding business processing for structural data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput;
Described at least one the storage server corresponding described second of described acquisition detects the Secondary resource consumption information of instruction, comprise: obtain each for carrying out the storage server of corresponding business processing for unstructured data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput.
Further,
Described structural data, comprising: numbers and symbols;
Described unstructured data, comprising: text, image and audio frequency.
Further,
Described target computation model, comprising: the computation model built based on any one the large data framework in following large data framework Hadoop, Hbase, Spark.
Second aspect, the invention provides a kind of testing server, comprising:
Setting unit, for being deployed at least one storage server by target computation model;
First acquiring unit, for obtaining target computation model; Obtain TPC-DS test suite; Obtain bigbench test suite;
Processing unit, instruction is detected for producing first according to described TPC-DS test suite, send first to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for structural data; Produce second according to described bigbench test suite and detect instruction, send second to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for unstructured data;
Second acquisition unit, detects the first resource consumption information of instruction for obtaining described at least one storage server corresponding described first; Obtain the Secondary resource consumption information that described at least one storage server corresponding described second detects instruction;
Storage unit, for storing described first resource consumption information; Store described Secondary resource consumption information.
Further,
Described storage unit, be further used for when described target computation model fails to provide first object data subset according to each first Structured Query Language (SQL) SQL query statement respectively, the first SQL query statement that record is corresponding and described target computation model fail to provide according to the first SQL query statement of correspondence the number of times of first object data subset; And, when described target computation model fails to provide the second target data subset according to each second SQL query statement respectively, the second SQL query statement that record is corresponding and described target computation model fail to provide according to the second SQL query statement of correspondence the number of times of the second target data subset.
Further,
Described second acquisition unit, for obtaining each for carrying out the storage server of corresponding business processing for structural data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput; Obtain each for carrying out the storage server of corresponding business processing for unstructured data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput.
The third aspect, the invention provides a kind of business platform, comprising:
As the testing server as described in arbitrary in above-mentioned second aspect and at least one storage server;
Described at least one storage server, carries out corresponding business processing for detecting instruction according to receive first for structural data; Detect instruction according to receive second and carry out corresponding business processing for unstructured data;
Each described storage server, comprising:
Monitoring resource device, detects resource consumption information corresponding when instruction carries out corresponding business processing for structural data for monitoring current storage server according to first; And/or, monitor current storage server and detect resource consumption information corresponding when instruction carries out corresponding business processing for unstructured data according to second.
Further,
Described at least one storage server, also comprises: processing unit, database; Described processing unit, for solution receive first detect that instruction receives to obtain at least one first Structured Query Language (SQL) SQL query statement and parsing second detect instruction to obtain at least one second SQL query statement; Described database, for providing first object data subset according to the first SQL query statement described in each to described testing server respectively; The second target data subset is provided to described testing server respectively according to the second SQL query statement described in each;
And/or,
Described monitoring resource device, for monitoring current storage server to obtain corresponding processor utilization, storage space consumption and network throughput.
The invention provides a kind of method of detection computations model, testing server and business platform, by target computation model is deployed at least one storage server, corresponding first resource consumption information when utilizing TPC-DS test suite to detect target computation model process structural data, and Secondary resource consumption information corresponding when utilizing bigbench test suite to detect target computation model process unstructured data; So, detect the resource consumption information that target computation model is corresponding respectively when processing dissimilar business datum, the sensing range for target computation model is more comprehensive, improves the accuracy of testing result.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the method flow diagram of a kind of detection computations model that one embodiment of the invention provides;
Fig. 2 is a kind of testing server that one embodiment of the invention provides;
Fig. 3 is a kind of business platform that one embodiment of the invention provides;
Fig. 4 is the another kind of business platform that one embodiment of the invention provides;
Fig. 5 is the method flow diagram utilizing business platform to realize detecting target computation model in one embodiment of the invention.
Embodiment
For making the object of the embodiment of the present invention, technical scheme and advantage clearly; below in conjunction with the accompanying drawing in the embodiment of the present invention; technical scheme in the embodiment of the present invention is clearly and completely described; obviously; described embodiment is the present invention's part embodiment, instead of whole embodiments, based on the embodiment in the present invention; the every other embodiment that those of ordinary skill in the art obtain under the prerequisite not making creative work, all belongs to the scope of protection of the invention.
As shown in Figure 1, embodiments provide a kind of method of detection computations model, the method can comprise the following steps:
S0: obtain target computation model, and target computation model is deployed at least one storage server;
S1: obtain TPC-DS test suite, produce first according to described TPC-DS test suite and detect instruction, send first to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for structural data;
S2: obtain the first resource consumption information that described at least one storage server corresponding described first detects instruction, and store described first resource consumption information;
S3: obtain bigbench test suite, produce second according to described bigbench test suite and detect instruction, send second to described target computation model and detect instruction, described second detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for unstructured data;
S4: obtain the Secondary resource consumption information that described at least one storage server corresponding described second detects instruction, and store described Secondary resource consumption information.
By target computation model is deployed at least one storage server, corresponding first resource consumption information when utilizing TPC-DS test suite to detect target computation model process structural data, and Secondary resource consumption information corresponding when utilizing bigbench test suite test target computation model process unstructured data; So, detect the resource consumption information that target computation model is corresponding respectively when processing dissimilar business datum, the sensing range for target computation model is more comprehensive, improves the accuracy of testing result.
Particularly, the project to be detected that target computation model is corresponding can comprise the project such as statistics, report generation, on-line equiries, data mining storing target data, large data sets, namely target computation model needs the business operation that can comprise corresponding above-mentioned test item according to the business processing detecting instruction execution, the data type used in the business operation process of corresponding above-mentioned test item can comprise the structural datas such as numeral and character, and the unstructured data such as document, image and audio frequency; Treatment effeciency corresponding respectively when same target computation model processes with unstructured data accordingly for structural data is not identical, and namely corresponding respectively resource consumption is not identical.
For example, when target computation model performs to database purchase business datum according to detection instruction, store to realize data because data base manipulation bivariate table carrys out logical expression business datum, the structural datas such as numeral and character can directly be stored in database; For unstructured data, unstructured data is needed to change accordingly, to generate the target data can carrying out logical expression with bivariate table, target data after conversion is stored in database, to realize the storage service for unstructured data, visible, when processing accordingly for unstructured data, will larger resource consumption be produced; Therefore, resource consumption information corresponding when utilizing multiple detection reference to detect the dissimilar business datum of target computation model process respectively, the sensing range for target computation model is more comprehensive, can improve the accuracy of testing result.
The explanation of value, structural data includes but not limited to numeral and character, and unstructured data includes but not limited to document, image and audio frequency, and such as, unstructured data can also comprise video.
Further, in one embodiment of the invention, the testing results such as the first resource consumption information of counter structure data and the Secondary resource consumption information of corresponding unstructured data being stored, facilitating user according to testing result for carrying out across comparison based on different large data frameworks between the different computation models completing same target business.
For example, said method is used to detect for the first object computation model built based on large data framework Hadoop with based on the second target computation model that large data framework Spark builds respectively, wherein first object computation model and the second object module complete same target business, such as, stores audio data, now, first object computation model and the second target computation model need data type to be processed to be unstructured data in the process completing target service, user is when contrasting the work efficiency of first object computation model and the second target computation model, only need the Secondary resource consumption information contrasting first object computation model and the second target computation model difference correspondence.
It should be noted that, for based on different large data frameworks and the multiple computation models completing same target business carry out across comparison time, should ensure that every configuration information of testing server and each storage server is completely the same, comprise each station server hard drive space corresponding respectively and preassembled aid etc.
Further, the database compatibility corresponding respectively due to the computation model based on the large data framework of difference is not identical yet, in order to the database compatibility realized for target computation model is corresponding detects, in a preferred embodiment of the invention,
Described first detects instruction carries at least one first Structured Query Language (SQL) SQL query statement, also comprises:
When described target computation model fails to provide first object data subset according to each first SQL query statement respectively, the first SQL query statement that record is corresponding and described target computation model fail to provide according to the first SQL query statement the number of times of first object data subset;
And,
Described second detects instruction carries at least one second SQL query statement, also comprises:
When described target computation model fails to provide the second target data subset according to each second SQL query statement respectively, the second SQL query statement that record is corresponding and described target computation model fail to provide according to the second SQL query statement the number of times of the second target data subset.
In one embodiment of the invention, the TPC-DS test suite based on TPC-DS test benchmark can comprise 99 SQL query, and the bigbench test suite based on bigbench test benchmark can comprise 30 SQL query; Utilize database that multiple SQL query statement query aim computation model is corresponding to obtain corresponding data subset respectively, when target computation model can not provide corresponding data subset according to the SQL query statement of correspondence, then illustrate that target computation model does not support this SQL query statement; On the one hand, the quantity of the SQL query statement can not supported according to target computation model determines the database compatibility that target computation model is corresponding; On the other hand, the SQL query statement do not supported for target computation model carries out record, convenient when being docked with external system by this target computation model, processes accordingly in advance for the SQL query statement that external system is corresponding.
Further, target computation model depends on storage server, detects target computation model and namely detects storage server corresponding to the target computation model resource consumption information corresponding when performing corresponding business; Particularly, in a preferred embodiment of the invention, described at least one the storage server corresponding described first of described acquisition detects the first resource consumption information of instruction, comprise: obtain each for carrying out the storage server of corresponding business processing for structural data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput;
Described at least one the storage server corresponding described second of described acquisition detects the Secondary resource consumption information of instruction, comprise: obtain each for carrying out the storage server of corresponding business processing for unstructured data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput.
The explanation of value, resource consumption information includes but not limited to processor utilization, storage space consumption and network throughput; Such as, hard disk IO can also be comprised.
What deserves to be explained is, the method of a kind of detection computations model that the embodiment of the present invention provides, can for build based on any large data framework computation model detect, such as, based on the computation model that any one the large data framework in following large data framework Hadoop, Hbase, Spark builds.
As described in Figure 2, embodiments provide a kind of testing server 20, comprising:
Setting unit 201, for being deployed at least one storage server by target computation model;
First acquiring unit 202, for obtaining target computation model; Obtain TPC-DS test suite; Obtain bigbench test suite;
Processing unit 203, instruction is detected for producing first according to described TPC-DS test suite, send first to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for structural data; Produce second according to described bigbench test suite and detect instruction, send second to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for unstructured data;
Second acquisition unit 204, detects the first resource consumption information of instruction for obtaining described at least one storage server corresponding described first; Obtain the Secondary resource consumption information that described at least one storage server corresponding described second detects instruction;
Storage unit 205, for storing described first resource consumption information; Store described Secondary resource consumption information.
In one embodiment of the invention, target computation model can be the computation model built based on any large data framework, for large data management business, such as, and the business such as cloud stores audio data, video data; Utilize and based on the test suite of different test benchmark, target computation model is detected, resource consumption information corresponding respectively during to obtain target computation model process dissimilar business datum, and store for testing result; On the one hand, the testing result of corresponding target computation model possesses higher accuracy, on the other hand, testing result is stored, when using the test of same testing server based on the large data framework of difference and the multiple computation models completing same target business detect, conveniently across comparison is carried out to testing result, after contrasting different testing results to make user, select suitable computation model to be applied in concrete business case.
Further, all there is certain compatibility issue in the database corresponding due to target computation model, namely different computation models is not identical to the degree of support of SQL query statement, therefore, in order to determine the database compatibility that target computation model is corresponding, in a preferred embodiment of the invention, described storage unit 205, be further used for when described target computation model fails to provide first object data subset according to each first Structured Query Language (SQL) SQL query statement respectively, the first SQL query statement that record is corresponding and described target computation model fail to provide according to the first SQL query statement of correspondence the number of times of first object data subset, and, when described target computation model fails to provide the second target data subset according to each second SQL query statement respectively, the second SQL query statement that record is corresponding and described target computation model fail to provide according to the second SQL query statement of correspondence the number of times of the second target data subset.
Particularly, in order to the resource consumption situation that the storage server relied on when realizing the business datum according to the process of testing result determination target computation model is dissimilar is corresponding, in a preferred embodiment of the invention, described second acquisition unit 204, for obtaining each for carrying out the storage server of corresponding business processing for structural data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput; Obtain each for carrying out the storage server of corresponding business processing for unstructured data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput.
As shown in Figure 3, embodiments provide a kind of business platform, comprising:
As the testing server 20 as described in arbitrary in above-described embodiment and at least one storage server 301;
Described at least one storage server 301, carries out corresponding business processing for detecting instruction according to receive first for structural data; Detect instruction according to receive second and carry out corresponding business processing for unstructured data;
Each described storage server 301, comprising:
Monitoring resource device 3011, detects resource consumption information corresponding when instruction carries out corresponding business processing for structural data for monitoring current storage server 301 according to first; And/or, monitor current storage server 301 and detect resource consumption information corresponding when instruction carries out corresponding business processing for unstructured data according to second.
In one embodiment of the invention, the target computation model based on large data framework is used for managing large data, therefore, target computation model can be deployed at least one storage server, utilize at least one the corresponding business datum of stores service management; In one mode in the cards, conveniently dissimilar data are managed respectively, the business datum that different stores service management is dissimilar can also be utilized, such as, structural data and unstructured data are stored in different storage servers respectively.
In one embodiment of the invention, monitoring resource device can comprise nmon performance analysis tool, and nmon performance analysis tool can monitor resource consumption information corresponding to current storage server in real time, and the resource consumption information of correspondence is sent to testing server.
Further, when target computation model realizes corresponding large data management business, need to depend on corresponding storage server, particularly, as shown in Figure 4, in one embodiment of the invention, described at least one storage server 301, also comprises: processing unit 3012, database 3013; Described processing unit 3012, for resolve receive first detect instruction to obtain at least one first Structured Query Language (SQL) SQL query statement; The second detection instruction that parsing receives is to obtain at least one second SQL query statement; Described database 3013, for providing first object data subset according to the first SQL query statement described in each to described testing server 20 respectively; The second target data subset is provided to described testing server 20 respectively according to the second SQL query statement described in each;
And/or,
Described monitoring resource device 311, for monitoring current storage server 31 to obtain corresponding processor utilization, storage space consumption and network throughput.
The content such as information interaction, implementation between each unit in said apparatus, due to the inventive method embodiment based on same design, particular content can see in the inventive method embodiment describe, repeat no more herein.
For making the object, technical solutions and advantages of the present invention clearly, below in conjunction with drawings and the specific embodiments, the present invention is described in further detail.
As shown in Figure 5, one embodiment of the invention, in conjunction with a kind of business platform that the embodiment of the present invention provides, the project be detected with detected target computation model is for storage service data instance, and the testing process for target computation model can comprise:
Step 501, in advance installation resource supervising device on each storage server.
In one embodiment of the invention, the monitoring resource device be arranged on each storage server can be used for the resource consumption information monitoring current storage server, such as, and processor utilization, storage space consumption and network throughput etc.
Particularly, monitoring resource device can comprise nmon performance analysis tool.
Step 502, testing server obtains target computation model, and is deployed at least one storage server by target computation model.
In one embodiment of the invention, target computation model is mainly used in large data management business; Can be the target computation model built based on any large data framework, such as, large data framework Hadoop, Hbase, Spark, Map/Reduce and HDFS etc.; In the embodiment of the present invention, test item for target computation model is storage service data, in order to ensure that target computation model possesses higher memory property, in general, target computation model can be deployed on multiple stage storage server to realize large operational data storage.
Step 503, testing server obtains TPC-DS test suite and bigbench test suite.
In one embodiment of the invention, the TPC-DS test suite and bigbench test suite that detect target computation model is configured in testing server, here, TPC-DS test suite and bigbench test suite are the testing softwares respectively based on TPC-DS test benchmark and bigbench test benchmark; Target computation model needs traffic data type to be processed to comprise structural data and unstructured data (comprising semi-structured data), corresponding resource consumption information is distinguished when using the TPC-DS test suite based on TPC-DS test benchmark and the bigbench test suite based on bigbench test benchmark to detect target computation model process different types of data respectively, sensing range for target computation model is more comprehensive, and testing result possesses higher accuracy.
Step 504, testing server produces the first detection instruction of carrying at least one the first SQL query statement according to described TPC-DS test suite, send first detect instruction to described target computation model.
In one embodiment of the invention, first detects instruction can carry 99 the first SQL query statements, be respectively used to database corresponding to query aim computation model to obtain corresponding first object data subset, first detects instruction indicating target computation model carries out corresponding stores processor business for structural data.
Particularly, the first SQL query statement can comprise:
select
l_returnflag,
l_linestatus,
sum(l_quantity)assum_qty,
sum(l_extendedprice)assum_base_price,
sum(l_extendedprice*(1-l_discount))assum_disc_price,
sum(l_extendedprice*(1-l_discount)*(1+l_tax))assum_charge,
avg(l_quantity)asavg_qty,
avg(l_extendedprice)asavg_price,
avg(l_discount)asavg_disc,
count(*)ascount_order
from
lineitem
where
l_shipdate<＝'1998-09-13'
groupby
l_returnflag,
l_linestatus
orderby
l_returnflag,
l_linestatuslimit10。
Step 505, at least one storage server carries out corresponding stores processor business for structural data, and the first resource consumption information of correspondence is sent to testing server, and, when the database that target computation model is corresponding supports the first corresponding SQL query statement, send corresponding first object data subset to testing server.
Step 506, testing server stores first resource consumption information, and, the first SQL query statement do not supported according to the first data subset record object computation model of correspondence and the quantity of the first SQL query statement do not supported.
Step 507, testing server produces the second detection instruction of carrying at least one the second SQL query statement according to described bigbench test suite, send second detect instruction to described target computation model.
In one embodiment of the invention, second detects instruction can carry 30 the second SQL query statements, be respectively used to database corresponding to query aim computation model to obtain the second corresponding data subset, second detects instruction indicating target computation model carries out corresponding stores processor business for unstructured data.
Particularly, the second SQL query statement can comprise:
Select
l_orderkey，
sum(l_extendedprice*(1-l_discount))asrevenue,
o_orderdate
o_shippriority
from
customer,
orders,
lineitem
where
c_mktsegmentlike'HOUSEHOLD％'
andc_custkey＝o_custkey
andl_orderkey＝o_orderkey
ando_orderdate<'1995-03-20'
andl_shipdate>'1995-03-20'
groupby
l_orderkey,
o_orderdate,
o_shippriority
orderby
revenuedesc,
o_orderdatelimit10。
Step 508, at least one storage server carries out corresponding stores processor business for unstructured data, and the Secondary resource consumption information of correspondence is sent to testing server, and, when the database that target computation model is corresponding supports the second corresponding SQL query statement, send the second corresponding target data subset to testing server.
The explanation of value be, conveniently target computation model is managed, different storage servers can be utilized to carry out corresponding stores processor business respectively for structural data and unstructured data, draw data by structural data and non-structural and be stored in respectively in different storage servers.
Step 209, testing server stores Secondary resource consumption information, and, the second SQL query statement do not supported according to the second data subset record object computation model of correspondence and the quantity of the second SQL query statement do not supported.
In one embodiment of the invention, in step 206 and step 209, the SQL query statement do not supported for target computation model respectively carries out record, facilitates user when being docked with external system by this target computation model, the SQL query statement that adjustment external system is corresponding; The quantity of the SQL query statement do not supported for target computation model and first resource consumption information, Secondary resource consumption store, the convenient information according to record carries out across comparison based on different large data frameworks between the multiple computation models completing same target business from other, selects suitable computation model to be applied to concrete business case to facilitate user according to comparing result.
Each embodiment of the present invention at least has following beneficial effect:
1, by target computation model is deployed at least one storage server, corresponding first resource consumption information when utilizing TPC-DS test suite to detect target computation model process structural data, and Secondary resource consumption information corresponding when utilizing bigbench test suite test target computation model process unstructured data; So, detect the resource consumption information that target computation model is corresponding respectively when processing dissimilar business datum, the sensing range for target computation model is more comprehensive, improves the accuracy of testing result.
2, by using the database that different SQL query statement query aim computation models is corresponding, and the SQL query statement do not supported of record object computation model and the quantity of SQL query statement do not supported thereof, the database compatibility of target computation model can be determined, and then the database compatibility realized for different computation models is corresponding respectively carries out across comparison, so that user selects suitable computation model to be applied to concrete business case in conjunction with practical business demand.
It should be noted that, in this article, the relational terms of such as first and second and so on is only used for an entity or operation to separate with another entity or operational zone, and not necessarily requires or imply the relation that there is any this reality between these entities or operation or sequentially.And, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element " being comprised a 〃 〃 〃 〃 〃 〃 " limited by statement, and be not precluded within process, method, article or the equipment comprising described key element and also there is other same factor.
Finally it should be noted that: the foregoing is only preferred embodiment of the present invention, only for illustration of technical scheme of the present invention, be not intended to limit protection scope of the present invention.All any amendments done within the spirit and principles in the present invention, equivalent replacement, improvement etc., be all included in protection scope of the present invention.
Claims (10)
1. a method for detection computations model, is characterized in that, comprising: obtain target computation model, and be deployed at least one storage server by target computation model; Also comprise:
Obtain TPC-DS test suite, produce first according to described TPC-DS test suite and detect instruction, send first to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for structural data;
Obtain the first resource consumption information that described at least one storage server corresponding described first detects instruction, and store described first resource consumption information;
Obtain bigbench test suite, produce second according to described bigbench test suite and detect instruction, send second to described target computation model and detect instruction, described second detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for unstructured data;
Obtain the Secondary resource consumption information that described at least one storage server corresponding described second detects instruction, and store described Secondary resource consumption information.
2. method according to claim 1, is characterized in that,
Described first detects instruction carries at least one first Structured Query Language (SQL) SQL query statement, also comprises:
When described target computation model fails to provide first object data subset according to each first SQL query statement respectively, the first SQL query statement that record is corresponding and described target computation model fail to provide according to the first SQL query statement the number of times of first object data subset;
And,
Described second detects instruction carries at least one second SQL query statement, also comprises:
When described target computation model fails to provide the second target data subset according to each second SQL query statement respectively, the second SQL query statement that record is corresponding and described target computation model fail to provide according to the second SQL query statement the number of times of the second target data subset.
3. method according to claim 1, is characterized in that,
Described at least one the storage server corresponding described first of described acquisition detects the first resource consumption information of instruction, comprise: obtain each for carrying out the storage server of corresponding business processing for structural data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput;
Described at least one the storage server corresponding described second of described acquisition detects the Secondary resource consumption information of instruction, comprise: obtain each for carrying out the storage server of corresponding business processing for unstructured data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput.
4. method according to claim 1, is characterized in that,
Described structural data, comprising: numbers and symbols;
Described unstructured data, comprising: text, image and audio frequency.
5., according to described method arbitrary in claim 1 to 5, it is characterized in that,
Described target computation model, comprising: the computation model built based on any one the large data framework in following large data framework Hadoop, Hbase, Spark.
6. a testing server, is characterized in that, comprising:
Setting unit, for being deployed at least one storage server by target computation model;
First acquiring unit, for obtaining target computation model; Obtain TPC-DS test suite; Obtain bigbench test suite;
Processing unit, instruction is detected for producing first according to described TPC-DS test suite, send first to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for structural data; Produce second according to described bigbench test suite and detect instruction, send second to described target computation model and detect instruction, described first detects instruction indicates at least one storage server corresponding to described target computation model to carry out corresponding business processing for unstructured data;
Second acquisition unit, detects the first resource consumption information of instruction for obtaining described at least one storage server corresponding described first; Obtain the Secondary resource consumption information that described at least one storage server corresponding described second detects instruction;
Storage unit, for storing described first resource consumption information; Store described Secondary resource consumption information.
7. testing server according to claim 6, is characterized in that,
Described storage unit, be further used for when described target computation model fails to provide first object data subset according to each first Structured Query Language (SQL) SQL query statement respectively, the first SQL query statement that record is corresponding and described target computation model fail to provide according to the first SQL query statement of correspondence the number of times of first object data subset; And, when described target computation model fails to provide the second target data subset according to each second SQL query statement respectively, the second SQL query statement that record is corresponding and described target computation model fail to provide according to the second SQL query statement of correspondence the number of times of the second target data subset.
8. testing server according to claim 7, is characterized in that,
Described second acquisition unit, for obtaining each for carrying out the storage server of corresponding business processing for structural data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput; Obtain each for carrying out the storage server of corresponding business processing for unstructured data when carrying out corresponding business processing, processor utilization corresponding respectively, storage space consumption and network throughput.
9. a business platform, is characterized in that, comprising:
As the testing server as described in arbitrary in the claims 6 to 8 and at least one storage server;
Described at least one storage server, carries out corresponding business processing for detecting instruction according to receive first for structural data; Detect instruction according to receive second and carry out corresponding business processing for unstructured data;
Each described storage server, comprising:
Monitoring resource device, detects resource consumption information corresponding when instruction carries out corresponding business processing for structural data for monitoring current storage server according to first; And/or, monitor current storage server and detect resource consumption information corresponding when instruction carries out corresponding business processing for unstructured data according to second.
10. business platform according to claim 9, is characterized in that,
Described at least one storage server, also comprises: processing unit, database; Described processing unit, for resolve receive first detect instruction to obtain at least one first Structured Query Language (SQL) SQL query statement, and, resolve receive second detect instruction to obtain at least one second SQL query statement; Described database, for providing first object data subset according to the first SQL query statement described in each to described testing server respectively; The second target data subset is provided to described testing server respectively according to the second SQL query statement described in each;
And/or,
Described monitoring resource device, for monitoring current storage server to obtain corresponding processor utilization, storage space consumption and network throughput.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510903291.XA CN105302730A (en) | 2015-12-09 | 2015-12-09 | Calculation model detection method, testing server and service platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510903291.XA CN105302730A (en) | 2015-12-09 | 2015-12-09 | Calculation model detection method, testing server and service platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105302730A true CN105302730A (en) | 2016-02-03 |
Family
ID=55200019
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510903291.XA Pending CN105302730A (en) | 2015-12-09 | 2015-12-09 | Calculation model detection method, testing server and service platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105302730A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273293A (en) * | 2017-06-16 | 2017-10-20 | 北京小度信息科技有限公司 | Big data system performance testing method, device and electronic equipment |
| CN108614761A (en) * | 2018-03-16 | 2018-10-02 | 重庆邮电大学 | Wisdom application system server performance demand computational methods based on business model |
| CN109299286A (en) * | 2018-09-28 | 2019-02-01 | 北京赛博贝斯数据科技有限责任公司 | The Knowledge Discovery Method and system of unstructured data |
| CN109358968A (en) * | 2018-10-08 | 2019-02-19 | 北京数码视讯软件技术发展有限公司 | A kind of server resource allocation method and device |
| CN109544341A (en) * | 2018-11-23 | 2019-03-29 | 中链科技有限公司 | Digitizing detection method and device based on block chain |
| CN109902027A (en) * | 2019-03-29 | 2019-06-18 | 梁俊杰 | A kind of TPC-DS test method and system convenient to carry out |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902769A (en) * | 2012-09-26 | 2013-01-30 | 曙光信息产业(北京)有限公司 | Database benchmark test system of cloud computing platform and method thereof |
| US20150019707A1 (en) * | 2013-07-09 | 2015-01-15 | Oracle International Corporation | Cloud services performance tuning and benchmarking |
| CN104866606A (en) * | 2015-06-02 | 2015-08-26 | 浙江师范大学 | MapReduce parallel big data text classification method |
-
2015
- 2015-12-09 CN CN201510903291.XA patent/CN105302730A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902769A (en) * | 2012-09-26 | 2013-01-30 | 曙光信息产业(北京)有限公司 | Database benchmark test system of cloud computing platform and method thereof |
| US20150019707A1 (en) * | 2013-07-09 | 2015-01-15 | Oracle International Corporation | Cloud services performance tuning and benchmarking |
| CN104866606A (en) * | 2015-06-02 | 2015-08-26 | 浙江师范大学 | MapReduce parallel big data text classification method |
Non-Patent Citations (2)
| Title |
|---|
| 周晓云等: "大数据评测基准的研发现状与趋势", 《计算机应用》 * |
| 姜春宇等: "大数据基准测试流程与测试工具", 《研究与开发》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273293A (en) * | 2017-06-16 | 2017-10-20 | 北京小度信息科技有限公司 | Big data system performance testing method, device and electronic equipment |
| CN107273293B (en) * | 2017-06-16 | 2020-09-11 | 北京星选科技有限公司 | Big data system performance test method and device and electronic equipment |
| CN108614761A (en) * | 2018-03-16 | 2018-10-02 | 重庆邮电大学 | Wisdom application system server performance demand computational methods based on business model |
| CN109299286A (en) * | 2018-09-28 | 2019-02-01 | 北京赛博贝斯数据科技有限责任公司 | The Knowledge Discovery Method and system of unstructured data |
| CN109358968A (en) * | 2018-10-08 | 2019-02-19 | 北京数码视讯软件技术发展有限公司 | A kind of server resource allocation method and device |
| CN109358968B (en) * | 2018-10-08 | 2021-06-25 | 北京数码视讯软件技术发展有限公司 | Server resource allocation method and device |
| CN109544341A (en) * | 2018-11-23 | 2019-03-29 | 中链科技有限公司 | Digitizing detection method and device based on block chain |
| CN109544341B (en) * | 2018-11-23 | 2023-07-07 | 苏州朗润创新知识产权运营有限公司 | Digital detection method and device based on block chain |
| CN109902027A (en) * | 2019-03-29 | 2019-06-18 | 梁俊杰 | A kind of TPC-DS test method and system convenient to carry out |
| CN109902027B (en) * | 2019-03-29 | 2020-05-29 | 梁俊杰 | TPC-DS testing method and system convenient to implement |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684352B (en) | Data analysis system, data analysis method, storage medium, and electronic device | |
| CN105302730A (en) | Calculation model detection method, testing server and service platform | |
| CN109902105B (en) | Data query system, method, device and storage medium for micro-service architecture | |
| KR20200106950A (en) | Dimensional context propagation techniques for optimizing SQL query plans | |
| CN107408114B (en) | Identifying join relationships based on transactional access patterns | |
| CN111709527A (en) | Operation and maintenance knowledge map library establishing method, device, equipment and storage medium | |
| CN110837520A (en) | Data processing method, platform and system | |
| CN113760891B (en) | Data table generation method, device, equipment and storage medium | |
| CN110543571A (en) | knowledge graph construction method and device for water conservancy informatization | |
| CN107016501A (en) | A kind of efficient industrial big data multidimensional analysis method | |
| CN112000773B (en) | Search engine technology-based data association relation mining method and application | |
| JP2010524060A (en) | Data merging in distributed computing | |
| US10002142B2 (en) | Method and apparatus for generating schema of non-relational database | |
| CN113312191A (en) | Data analysis method, device, equipment and storage medium | |
| CN113468019A (en) | Hbase-based index monitoring method, device, equipment and storage medium | |
| CN112579578A (en) | Metadata-based data quality management method, device and system and server | |
| CN112258244B (en) | Method, device, equipment and storage medium for determining task to which target object belongs | |
| CN111078761A (en) | Data probing method, device, equipment and storage medium | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| WO2015196885A1 (en) | Method and apparatus for acquiring and storing performance data of cloud computing system | |
| CN114637811A (en) | Data table entity relation graph generation method, device, equipment and storage medium | |
| CN115080570A (en) | Graph database-based field blood relationship processing method and system | |
| CN112148719B (en) | Data processing query method and device based on OLAP pre-calculation model | |
| CN113722296A (en) | Agricultural information processing method and device, electronic equipment and storage medium | |
| US20150134660A1 (en) | Data clustering system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160203 |