CN102915303B - A kind of method and apparatus of ETL test - Google Patents
A kind of method and apparatus of ETL test Download PDFInfo
- Publication number
- CN102915303B CN102915303B CN201110218915.6A CN201110218915A CN102915303B CN 102915303 B CN102915303 B CN 102915303B CN 201110218915 A CN201110218915 A CN 201110218915A CN 102915303 B CN102915303 B CN 102915303B
- Authority
- CN
- China
- Prior art keywords
- test
- subobject
- etl
- tested object
- split
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
This application provides the method and apparatus that a kind of ETL tests, wherein, described method comprises: according to presetting rule, ETL tested object is split as test subobject; Test one by one for described test subobject.The application can reduce the complexity of ETL tested object, improves testing efficiency.
Description
Technical field
The application relates to the technical field of Analysis of Data Warehouse, particularly relates to the method and apparatus that a kind of ETL tests.
Background technology
Data warehouse is an independently data environment, needs to be imported to data warehouse by the data storage medium of data from Transaction Processing environment, external data source and off line by extraction process.Its object sets up structurized data space, by the data separating in different pieces of information source out, forms unified, effective data set, and finally process, be integrated into required data.
ETL is the process of data pick-up (Extract), cleaning (Cleaning), conversion (Transform), loading (Load).Be the important ring building data warehouse, user extracts required data from data source, through data cleansing, finally according to the data warehouse model pre-defined, by Data import in data warehouse.In the test process of ETL, tested object is formed by multiple task scheduling usually, or, dispatched by individual task and formed, but include one or more hundred row level codes in usual individual task.
Adopt prior art, the test process of ETL can comprise the following steps usually:
The Data Matching (mapping) of step 1, source data and target data is analyzed:
Analyze the source table relied on, object table and mutual dependence;
Step 2, design test case:
Storing process is regarded as an entirety, pay close attention to input and output, according to business function design test case;
Step 3, data encasement:
Prepare the data of the source table of each dependence;
Step 4, carry out result verification:
Result verification is carried out with the comparison that object table two is shown by source table.
In the test process of ETL, tested object is made up of multiple task scheduling, or individual task is one or more hundred row levels or thousand row level codes composition, and its data volume is very huge, and interrelated between the table of multiple source.Adopt prior art, tested object is integrally tested, test process will be made very complicated, task amount is huge, and one of them little details goes wrong, and whole test result will be made mistakes, just need to re-start test, thus make the testing efficiency of whole ETL very low.Therefore, the technical matters needing those skilled in the art urgently to solve at present is exactly: provide a kind of ETL testing mechanism, in order to reduce the complexity of ETL tested object, improves testing efficiency.
Summary of the invention
Technical problems to be solved in this application are to provide a kind of method that ETL tests, and based on the application, can reduce the complexity of ETL tested object, improve testing efficiency.
Accordingly, present invention also provides the device of ETL test, in order to ensure said method implementation and application in practice.
In order to solve the problem, this application discloses a kind of ETL method of testing, comprising:
According to presetting rule, ETL tested object is split as test subobject;
Test one by one for described test subobject.
Preferably, described step ETL tested object being split as test subobject according to presetting rule comprises:
Obtain the number of test subobject detachable in ETL tested object;
According to described detachable test subobject number, ETL tested object is split as the test subobject of corresponding number.
Preferably, the step of the number of detachable in described acquisition ETL tested object test subobject comprises further:
Split for each business comprised in ETL tested object, obtain detachable test subobject number;
Or,
Split for each function comprised in ETL tested object, obtain detachable test subobject number.
Preferably, the step of the number of detachable in described acquisition ETL tested object test subobject comprises:
Obtain temporary table and the object table information of storing process in described ETL tested object;
Calculate the quantity sum of described temporary table and an object table, as detachable test subobject number.
Preferably, described according to detachable test subobject number, step ETL tested object being split as test subobject comprises:
In the code that ETL tested object is corresponding, extract the code segment inserting identical temporary table;
Merge multiple code segments of the identical temporary table of described insertion, composition test subobject.
Preferably, the code that described ETL tested object is corresponding is mate the code segment of Insert to Select in the code of ETL tested object, and described test subobject is the SQL script of Insert to Select code segment composition.
Preferably, described step of carrying out one by one testing for test subobject comprises:
For described test subobject setup test use-case and test data;
Perform the test case adopting described test data, obtain test result.
Preferably, the described step for test subobject setup test use-case and test data comprises:
Test case is write according to test purpose;
Obtain the source table information that current test subobject relies on.
Preferably, described method, also comprises:
Encapsulate described test subobject, and batch performs test case corresponding to described test subobject, obtains the test result of described ETL tested object.
Present invention also provides the device that a kind of ETL tests, comprising:
Split module, for ETL tested object being split as test subobject according to presetting rule;
Test module, for testing one by one for described test subobject.
Compared with prior art, the application has the following advantages:
In terms of existing technologies, the application, ETL tested object is split as test subobject according to presetting rule, then tests for described test subobject one by one, complicated tested object is simplified, reduces testing complex degree, improve testing efficiency.
Test for test subobject one by one the range shorter that single is tested one by one, precise positioning test specification, has mutual independence.Each test subobject has independence, can test separately, and relying on point at test subobject can drive piles.Because single test subobject test specification is little and clear and definite, simple, once after going wrong, energy quick position is to problem place.
Simultaneously when test data prepares, can avoid making the table data in most source, and according to business, carry out the structure of simple temporary table data, reduce the cost of data configuration.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of a kind of ETL method of testing embodiment 1 of the application;
Fig. 2 is the process flow diagram of a kind of ETL method of testing embodiment 2 of the application;
Fig. 3 is the structured flowchart of a kind of ETL proving installation embodiment 1 of the application;
Fig. 4 is the structured flowchart of a kind of ETL proving installation embodiment 2 of the application.
Embodiment
For enabling above-mentioned purpose, the feature and advantage of the application more become apparent, below in conjunction with the drawings and specific embodiments, the application is described in further detail.
For making those skilled in the art understand the application better, with general project situation, the basic procedure that ETL tests is described below.ETL test generally comprises following link: demand analysis, test analysis, normal data structure, method of testing, Test Sample Design, test result checking and issue.
(1), demand analysis
Need to be familiar with operation flow and business rule in this link.
(2), test analysis
This link needs to determine each test point.ETL test mainly comprises ETL routine inspection and service logic inspection.
Routine inspection comprises:
1, whether ETL script has run-time error, script runtime (seeing executive plan);
2, the error handling mechanism of ETL script whether complete (code review);
3, whether ETL script supports rollback.
Service logic inspection comprises:
1, the inspection of data volume.Whether unanimously with expection check record number;
2, uniqueness inspection, mainly checks whether major key repeats (whether cookie_id, member_id repeat);
3, service fields conversion Correctness checking, check that whether index calculate is correct, can by sampling check, source table and object table are respectively got some records, judge field mappings whether correct (map field), also have index calculate whether correct (index calculate field);
4, randomness checking (get several data at random, see if there is mess code, abnormal data etc.).
Test emphasis and the test specification of Water demand ETL test is gone back, for the key business in project, using complex logic part as test emphasis, and for tested object by each index Design use-case in test analysis.Also need analysis source to show simultaneously, put in order source table between incidence relation, evaluating objects table, analyze according to demand source table and object table and between mapping relation, parse the data flow diagram of business.
(3), method of testing
Method of testing is the test based on inquiry, and expected results represents based on sql, accomplishes data variation, and result is constant.Be convenient in addition return, Test Strategy adopts incremental testing namely progressively to submit test to.Need in this process to use temporary table in a large number.
(4), standard data set builds
Standard data set builds point two aspects, and an aspect is the data directly extracted on line, and another aspect makes abnormal data with script.
DataBase combining (dblink) is utilized to extract data on line.When extracting data on line, it is noted that the comprehensive of test data.Namely test data covers comprehensively.Such as sex (sex) field, when extracting data on line, need to be drawn into man (male), female (female) situation, and can not be only male or female, such test data will lack.When data pick-up is carried out for related table, first can extract master meter, then according to the data of master meter extraction sublist with good conditionsi.
Make abnormal data, abnormal data can be considered from several aspect below: field type, field length, null value, service exception value, unique constraint value.
(5), Test Sample Design
Test case can design separately, and the thought of scheduling also can be adopted to design, when adopting dispatching method to design, and can the multiple use-case of one-time authentication, in addition also convenient recurrence.
As example, total script of adjusting is:
Test case 1 is:
(6), test result checking
Test result checking comprises two steps:
The first step first verifies that whether record number is consistent, if record number is inconsistent, then necessarily has problems, checks problem, find out reason.Second step is when recording number and being consistent, and whether check the value is correct.
Two kinds of methods specifically can be adopted to realize:
Method 1: use minus
SELECT*FROMtarget_a
MINUS
SELECT*FROMtest_map；
It should be noted that the position must exchanging 2 tables herein compares.Minus function, when comparing, is as the criterion with first table, finds out in first table and tables inconsistent place with second.That is: find out in sumdt0 table and show inconsistent result with map.
Method 2: write script and verify.
Be more than the general process that in prior art, ETL test adopts, in this process, there is following problem:
1, there is branch and omit risk, above test process regards one or more storing process as a black box, and that cannot pay close attention to inside realizes logic, may there is branch and omit.Have ready conditions in such as storing process and judge ifelse, not understanding inner realization cannot cover all branches completely.
2, data encasement is complicated, ETL storing process generally can relate to one to n source table, when carrying out standard data set structure, need the disposable data getting out n table, the data of this n table just may be used in last several step of storing process, and process is long, if having main foreign key relationship between the table of source, the complexity of data encasement can be increased further, and easily make mistakes.
3, investigate problem hard, tested object is regarded as an entirety, once there is Bug, be difficult to locate problematic code in the very first time, need debugging breakpoints, the investigation of a line line code, if code line is many, investigation is got up can be more time-consuming.
4, cannot test for test emphasis, some key projects and complex logic part is had to be need as test emphasis in ETL test, such as get the record number of top100, sequence is a test emphasis, a storing process in the end step carries out the top100 sequence of rownum, adopt existing method of testing, then need to make data from source.
5, Bug verifies that scope is exaggerated, and adopts prior art to test, and after Bug is repaired in exploitation, tester cannot know which code of concrete modification, and be integrated testability because of what adopt, only cannot verify for the part of amendment, but need again to return, be exaggerated test specification.
For the problems referred to above, the core idea of the creationary proposition the application of applicant is that ETL tested object is split as test subobject according to presetting rule, then tests one by one for described test subobject, thus reduce the complexity of ETL tested object, improve testing efficiency.
With reference to Fig. 1, show the process flow diagram of the embodiment of the method 1 of the application ETL test, specifically can comprise the following steps:
Step 101, according to presetting rule, ETL tested object is split as test subobject.
ETL tested object is split as test subobject according to presetting rule, complicated tested object is simplified, reduces testing complex degree.ETL tested object is in the following example:
In a kind of preferred embodiment of the application, described step 101 can comprise:
The number of test subobject detachable in sub-step S11, acquisition ETL tested object.This step is by analyzing tested object, designs the number of detachable test subobject.
In a kind of preferred embodiment of the application, described sub-step S11 can comprise:
Sub-step S11-1, to split for each business comprised in ETL tested object, obtain detachable test subobject number;
Or,
Sub-step S11-2, to split for each function comprised in ETL tested object, obtain detachable test subobject number.
ETL tested object has been the process that paired data extracts, conversion is processed and loaded, wherein contain multiple modules that can realize its business or function, design test subobject number is very crucial, the number of test subobject is not The more the better, the function segmentation can carried out for tested object in concrete realization, each module split is called a test subobject; Or for the business separation that tested object carries out, each module split is called a test subobject.Each test subobject correspondence realizes certain function or business.
Such as, task A is made up of 3 sub-tasks, and each subtask correspondence completes a function, now the test of task A can be split into 3 test subobjects.
Sub-step S12, according to described detachable test subobject number, ETL tested object is split as the test subobject of corresponding number.
In a kind of preferred embodiment of the application, described sub-step S12 can comprise:
Sub-step S12-1, in the code that ETL tested object is corresponding, extract and insert the code segment of identical temporary table;
Sub-step S12-2, merge multiple code segments of the identical temporary table of described insertion, composition test subobject.
In concrete realization, the code that described ETL tested object is corresponding can be mate the code segment of Insert to Select in the code of ETL tested object, and described test subobject can be the SQL script of Insert to Select code segment composition.
As the example in step 101, according to business or function, it is split, three test subobjects can be divided into, using the code segment of coupling Insert to Select as code corresponding to ETL tested object, in the code of correspondence, there is the temporary table that two different: temp0, a temp1 and object table target01, code segment corresponding for each temporary table is extracted, again each code segment is merged, be referred to as a test subobject, be specially and the code segment of inserttemp0 table multiple in code is merged, be called test subobject 1, the code segment of inserttemp1 table multiple in code is merged, be called test subobject 2, object table is inserted in remaining code segment, these code segments are merged, be called test subobject 3.Each test subobject shows as the SQL script of insert to select code segment composition, as follows.
Test subobject 1:
insertintotable_temp0()
selecta.*fromtable01a；
Test subobject 2:
insertintotable_temp1()
selecta.*fromtable_temp0a；
Test subobject 3:
insertintotable_target01
selecta.*fromtable_temp1；
In actual applications, if ETL tested object more complicated, after once splitting, each test subobject is still huger, then can carry out splitting again or more frequently to test subobject, the application is not restricted at this.Such as, ETL tested object is split as test subobject 1, test subobject 2 and test subobject 3, again test subobject 1 can be split as test subobject 1.1, test subobject 1.2 and test subobject 1.3.Carry out it should be noted that when splitting when to exist between certain several test subobject be depth dependency, need these test subobjects integrally to test in this case.
Step 102, to test one by one for described test subobject.
In a kind of preferred embodiment of the application, step 102 can comprise:
Sub-step S31, for described test subobject setup test use-case and test data.Test case and test data have been used to the test of pairwise testing subobject.
In a kind of preferred embodiment of the application, sub-step S31 can comprise:
Sub-step S31-1, write test case according to test purpose.Test purpose mainly comprises routine inspection and service logic inspection.Such as, uniqueness inspection, service fields conversion Correctness checking, inspection of data volume etc.Can write test case by UE instrument or notepad in concrete realization, test case represents with the form of sql.
Such as: the test case of design test subobject 1:
Test case 1: uniqueness inspection
Selectcount(0)fromtable_temp0agroupbya.idhavingcount(0)＞1；
Test case 2: index checking
Insertintoresult
selectid，sum(a.gmv)asgmvfromtable01agroupbya.id
Sub-step S31-2, obtain the source table information that current test subobject relies on.In concrete realization, directly can carry out data encasement by sql statement.As insertintotestable (a, b) values (a, b); Or, storing process can be write or java program realizes, as: writing Di Kaer collection, assignment etc. that circulation realizes certain several field, generate SQL statement, realize data encasement, the data herein prepared are only used for the test data of single test subobject, construct fairly simple.
The source table such as analyzing the dependence of test subobject 1 is table01, so data encasement is for table01 data encasement script:
Insertintotable01(id，gmv)values(1，200)；
Insertintotable01(id，gmv)values(2，300)；
The test case of sub-step S32, the described test data of execution employing, obtains test result.Described test data is updated in test case, implementation of test cases, the operation result of checkout use-case, completes the test of corresponding test subobject.
In concrete realization, the operation result of checkout use-case can use the following two kinds method:
1, directly comparison.Check that whether source table a is equal with the value of each field of object table b by the minus function of database.As: Selectcol1, col2fromaminusselectcol1, col2fromb.
2, storing process is write or java program realizes.As: the record number of for circulation source table a and object table b, field is compared one by one.
As above example, implementation of test cases 1 in sql environment:
SQL＞Selectcount(0)fromtable_temp0agroupbya.idhavingcount(0)＞1；
Check that operation result: Ifcount (0) <=0 test case is passed through.
Next, implementation of test cases 2 in sql environment:
SQL＞Insertintoresult
selectid，sum(a.gmv)asgmvfromtable01agroupbya.id
Check operation result:
selecta.id，a.gmvfromresulta
minus
selectb.id，b.gmvfromtable_temp0b
So just, obtain the test result of test subobject, the test result that just can obtain ETL tested object is detected to all test subobjects.Each tested object is separate, once go wrong, the problem that first finds, on which test subobject, then retests the single test subobject gone wrong; To test the functional module of emphasis, can extract and test subobject accordingly, then test for test subobject.
With reference to Fig. 2, show the process flow diagram of the embodiment of the method 2 of the application ETL test, specifically can comprise the following steps:
Step 201, according to presetting rule, ETL tested object is split as test subobject;
In a kind of preferred embodiment of the application, step 201 can comprise:
The number of test subobject detachable in sub-step S41, acquisition ETL tested object;
Sub-step S42, according to described detachable test subobject number, ETL tested object is split as the test subobject of corresponding number.
In a kind of preferred embodiment of the application, sub-step S41 can comprise:
Sub-step S41-1, the temporary table obtaining storing process in described ETL tested object and object table information;
As above the ETL tested object of example:
Wherein, comprise two temporary table: temp0 and temp1, an object table target01.
Sub-step S41-2, calculate the quantity sum of described temporary table and an object table, as detachable test subobject number.
As above example, in tested object, the number of temp table is 2, then temporary table quantity adds that the quantity of 1 object table is 3, therefore the number of detachable test subobject number is 3.
Step 202, to test one by one for described test subobject.
Step 203, encapsulate described test subobject, and batch performs test case corresponding to described test subobject, obtains the test result of described ETL tested object.
Can carry out integrated testability again after the test of test subobject and also be called integration testing, be encapsulated into by the test subobject generated according to above-mentioned method for splitting in a sql file, in database environment, batch performs the test case of sql composition, completes integrated testability.
In sum, this application provides a kind of method that ETL tests, by ETL tested object is split as test subobject according to presetting rule, then test one by one for described test subobject, complicated tested object is simplified, reduces testing complex degree, improve testing efficiency.
Test one by one for test subobject one by one, this not only can reduce to obtain the scope of single test, but also can precise positioning test specification, and test subobject has mutual independence in tested process.And each test subobject has independence, can test separately, relying on point at test subobject can drive piles.Single test subobject test specification is little and clear and definite, simple, once after going wrong, energy quick position is to problem place.
Simultaneously when test data prepares, can avoid making the table data in most source, and according to business, carry out the structure of simple temporary table data, reduce the cost of data configuration.
ETL field tests is a newer field, and the method for testing for this field is less.Present inventor carries out the research of method of testing for the object of ETL field tests, first measurand is split, and uses the overall thought with splitting subobject to test.The application is highly suitable for the feature of ETL domain object, and the source table namely related to is many, interrelated between the table of source; Data volume is large, and millions is T level even; Code line level is hundred grades, thousand row levels etc.The application proposes a kind of new solution to testing thought and test mode and test concept.
It should be noted that, for embodiment of the method, in order to simple description, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the application is not by the restriction of described sequence of movement, because according to the application, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and involved action and module might not be that the application is necessary.
With reference to Fig. 3, show the structured flowchart of the device embodiment 1 of the application ETL test, specifically can comprise with lower module:
Split module 301, for ETL tested object being split as test subobject according to presetting rule;
Test module 302, for testing one by one for described test subobject.
In a kind of preferred embodiment of the application, splitting module 301 can comprise:
Number obtains submodule, for obtaining the number of test subobject detachable in ETL tested object;
Tested object splits submodule, for according to described detachable test subobject number, ETL tested object is split as the test subobject of corresponding number.
In a kind of preferred embodiment of the application, described number obtains submodule and may further include: business separation submodule, for splitting for each business comprised in ETL tested object, obtains detachable test subobject number;
Or,
Function segmentation submodule, for splitting for each function comprised in ETL tested object, obtains detachable test subobject number.
In a kind of preferred embodiment of the application, described tested object splits submodule and can comprise:
Code segment extracts submodule, in the code that ETL tested object is corresponding, extracts the code segment inserting identical temporary table;
Code segment merges submodule, for merging multiple code segments of the identical temporary table of described insertion, and composition test subobject.
In concrete realization, the code that described ETL tested object is corresponding is mate the code segment of Insert to Select in the code of ETL tested object, and described test subobject is the SQL script of Insert to Select code segment composition.
In a kind of preferred embodiment of the application, described test module 302 can comprise:
Test prepares submodule, for for described test subobject setup test use-case and test data;
Test case implementation sub-module, for performing the test case adopting described test data, obtains test result.
In concrete realization, described test prepares submodule and can comprise:
Test case writes submodule, for writing test case according to test purpose;
Source table acquisition of information submodule, for obtaining the source table information that current test subobject relies on.
With reference to Fig. 4, show the structured flowchart of the device embodiment 2 of the application ETL test, specifically can comprise with lower module:
Split module 401, for ETL tested object being split as test subobject according to presetting rule;
Test module 402, for testing one by one for described test subobject;
Test result acquisition module 403, for encapsulating described test subobject, and batch performs test case corresponding to described test subobject, obtains the test result of described ETL tested object.
In a kind of preferred embodiment of the application, described fractionation module 401 can comprise:
Number obtains submodule, for obtaining the number of test subobject detachable in ETL tested object;
Tested object splits submodule, for according to described detachable test subobject number, ETL tested object is split as the test subobject of corresponding number.
In concrete realization, described number obtains submodule and can comprise:
Acquisition of information submodule, for obtaining temporary table and the object table information of storing process in described ETL tested object;
Number calculating sub module, for calculating the quantity sum of described temporary table and an object table, as detachable test subobject number.
Because described device embodiment is substantially corresponding to the embodiment of the method shown in earlier figures 1 and Fig. 2, therefore not detailed part in the description of the present embodiment, see the related description in previous embodiment, just can not repeat at this.
The application can be used in numerous general or special purpose computing system environment or configuration.Such as: personal computer, server computer, handheld device or portable set, laptop device, multicomputer system, system, set top box, programmable consumer-elcetronics devices, network PC, small-size computer, mainframe computer, the distributed computing environment comprising above any system or equipment etc. based on microprocessor.
The application can describe in the general context of computer executable instructions, such as program module.Usually, program module comprises the routine, program, object, assembly, data structure etc. that perform particular task or realize particular abstract data type.Also can put into practice the application in a distributed computing environment, in these distributed computing environment, be executed the task by the remote processing devices be connected by communication network.In a distributed computing environment, program module can be arranged in the local and remote computer-readable storage medium comprising memory device.
Finally, also it should be noted that, in this article, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical element.
Above to the method for a kind of ETL test that the application provides, and, a kind of device of ETL test is described in detail, apply specific case herein to set forth the principle of the application and embodiment, the explanation of above embodiment is just for helping method and the core concept thereof of understanding the application; Meanwhile, for one of ordinary skill in the art, according to the thought of the application, all will change in specific embodiments and applications, in sum, this description should not be construed as the restriction to the application.
Claims (8)
1. a method for ETL test, is characterized in that, comprising:
According to presetting rule, ETL tested object is split as test subobject, wherein, described ETL tested object has been the process that paired data extracts, conversion is processed and loaded, and contains multiple modules that can realize its business or function; Described test subobject correspondence realizes certain function or business;
Test one by one for described test subobject;
Encapsulate described test subobject, and batch performs test case corresponding to described test subobject, obtains the test result of described ETL tested object;
Wherein, described step ETL tested object being split as test subobject according to presetting rule comprises:
Obtain the number of test subobject detachable in ETL tested object;
According to described detachable test subobject number, ETL tested object is split as the test subobject of corresponding number.
2. method according to claim 1, is characterized in that, the step of the number of test subobject detachable in described acquisition ETL tested object comprises further:
Split for each business comprised in ETL tested object, obtain detachable test subobject number;
Or,
Split for each function comprised in ETL tested object, obtain detachable test subobject number.
3. method according to claim 1, is characterized in that, the step of the number of test subobject detachable in described acquisition ETL tested object comprises:
Obtain temporary table and the object table information of storing process in described ETL tested object;
Calculate the quantity sum of described temporary table and an object table, as detachable test subobject number.
4. according to the method in claim 2 or 3, it is characterized in that, described according to detachable test subobject number, step ETL tested object being split as test subobject comprises:
In the code that ETL tested object is corresponding, extract the code segment inserting identical temporary table;
Merge multiple code segments of the identical temporary table of described insertion, composition test subobject.
5. method according to claim 4, it is characterized in that, the code that described ETL tested object is corresponding is mate the code segment of Insert to Select in the code of ETL tested object, and described test subobject is the SQL script of Insert to Select code segment composition.
6. method according to claim 1, is characterized in that, described step of carrying out one by one testing for test subobject comprises:
For described test subobject setup test use-case and test data;
Perform the test case adopting described test data, obtain test result.
7. method according to claim 6, is characterized in that, the described step for test subobject setup test use-case and test data comprises:
Test case is write according to test purpose;
Obtain the source table information that current test subobject relies on.
8. a device for ETL test, is characterized in that, comprising:
Split module, for ETL tested object being split as test subobject according to presetting rule, wherein, described ETL tested object has been the process that paired data extracts, conversion is processed and loaded, and contains multiple modules that can realize its business or function; Described test subobject correspondence realizes certain function or business;
Test module, for testing one by one for described test subobject;
Test result acquisition module, for encapsulating described test subobject, and batch performs test case corresponding to described test subobject, obtains the test result of described ETL tested object;
Wherein, split module to comprise:
Number obtains submodule, for obtaining the number of test subobject detachable in ETL tested object;
Tested object splits submodule, for according to described detachable test subobject number, ETL tested object is split as the test subobject of corresponding number.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110218915.6A CN102915303B (en) | 2011-08-01 | 2011-08-01 | A kind of method and apparatus of ETL test |
| HK13103671.5A HK1176710A1 (en) | 2011-08-01 | 2013-03-25 | Method and device for an extraction- transformation-loading test etl |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110218915.6A CN102915303B (en) | 2011-08-01 | 2011-08-01 | A kind of method and apparatus of ETL test |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102915303A CN102915303A (en) | 2013-02-06 |
| CN102915303B true CN102915303B (en) | 2016-04-20 |
Family
ID=47613674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110218915.6A Active CN102915303B (en) | 2011-08-01 | 2011-08-01 | A kind of method and apparatus of ETL test |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN102915303B (en) |
| HK (1) | HK1176710A1 (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10296448B2 (en) | 2014-11-19 | 2019-05-21 | International Business Machines Corporation | Correlating test results variations with business requirements |
| CN104614601B (en) * | 2014-12-26 | 2019-02-15 | 惠州Tcl移动通信有限公司 | A kind of terminal fault localization method, apparatus and system |
| CN106874290B (en) * | 2015-12-11 | 2020-08-04 | 阿里巴巴集团控股有限公司 | Data cleaning method and equipment |
| US10929281B1 (en) * | 2016-05-20 | 2021-02-23 | Jpmorgan Chase Bank, N.A. | Systems and methods for testing of data transformations |
| CN107678790B (en) | 2016-07-29 | 2020-05-08 | 华为技术有限公司 | Flow calculation method, device and system |
| CN106771453B (en) * | 2016-11-16 | 2020-10-02 | 广州视源电子科技股份有限公司 | Testing method and device for oscilloscope |
| CN108153662A (en) * | 2016-12-05 | 2018-06-12 | 北京国双科技有限公司 | A kind of data test method and data testing system |
| CN107491388A (en) * | 2017-07-28 | 2017-12-19 | 深圳市元征科技股份有限公司 | Bug method and device in a kind of positioning program code |
| CN113934786B (en) * | 2021-09-29 | 2023-09-08 | 浪潮卓数大数据产业发展有限公司 | Implementation method for constructing unified ETL |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101105793A (en) * | 2006-07-11 | 2008-01-16 | 阿里巴巴公司 | Data processing method and system of data library |
| CN101242617A (en) * | 2008-03-04 | 2008-08-13 | 中兴通讯股份有限公司 | A testing system and method for original performance statistical data |
| CN102063372A (en) * | 2010-12-30 | 2011-05-18 | 浪潮集团山东通用软件有限公司 | Main key driven modularized automated test method |
| CN102129425A (en) * | 2010-01-20 | 2011-07-20 | 阿里巴巴集团控股有限公司 | Method and device for accessing large-object set table in data warehouse |
-
2011
- 2011-08-01 CN CN201110218915.6A patent/CN102915303B/en active Active
-
2013
- 2013-03-25 HK HK13103671.5A patent/HK1176710A1/en unknown
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101105793A (en) * | 2006-07-11 | 2008-01-16 | 阿里巴巴公司 | Data processing method and system of data library |
| CN101242617A (en) * | 2008-03-04 | 2008-08-13 | 中兴通讯股份有限公司 | A testing system and method for original performance statistical data |
| CN102129425A (en) * | 2010-01-20 | 2011-07-20 | 阿里巴巴集团控股有限公司 | Method and device for accessing large-object set table in data warehouse |
| CN102063372A (en) * | 2010-12-30 | 2011-05-18 | 浪潮集团山东通用软件有限公司 | Main key driven modularized automated test method |
Non-Patent Citations (2)
| Title |
|---|
| 《基于分布式数据仓库技术的ETL系统的研究与应用》;花海洋等;《微计算机信息》;20061231;第22卷(第10-3期);正文第144-147页 * |
| 《集中式系统的分布式通用查询框架设计》;白剑斐等;《计算机工程》;20101031;第36卷(第20期);正文第71-73页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| HK1176710A1 (en) | 2013-08-02 |
| CN102915303A (en) | 2013-02-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102915303B (en) | A kind of method and apparatus of ETL test | |
| US9804946B2 (en) | System and method for providing automated computer language translation and verification | |
| CN107895286B (en) | Claim amount determining method and device, storage medium and electronic equipment | |
| US9043759B1 (en) | System and method for generating software unit tests simultaneously with API documentation | |
| US9280442B1 (en) | System and method for generating coverage reports for software unit tests | |
| Mirzaaghaei et al. | Supporting test suite evolution through test case adaptation | |
| US8972938B2 (en) | Determining functional design/requirements coverage of a computer code | |
| CN102567166B (en) | Testing method and testing system of graphics card | |
| Falke et al. | The bounded model checker LLBMC | |
| Grafberger et al. | Lightweight inspection of data preprocessing in native machine learning pipelines | |
| US20110271253A1 (en) | Enhancing functional tests coverage using traceability and static analysis | |
| CN104407980A (en) | Mobile application automated testing device and method | |
| US10013335B2 (en) | Data flow analysis in processor trace logs using compiler-type information method and apparatus | |
| CN103678115B (en) | The apparatus and method of the position of source code mistake are detected in mixed mode program | |
| Hammad et al. | Automatically identifying changes that impact code-to-design traceability during evolution | |
| CN103309793A (en) | Method and system for ranking analysis tools | |
| CN112765017A (en) | Data query performance test method and device based on MySQL database | |
| Boussaa et al. | Leveraging metamorphic testing to automatically detect inconsistencies in code generator families | |
| Zhang et al. | A study of persistent memory bugs in the linux kernel | |
| Hammad et al. | Automatically identifying changes that impact code-to-design traceability | |
| Nelson et al. | Cooperative mode: Comparative storage metadata verification applied to the Xbox 360 | |
| US20120124428A1 (en) | Method and system for testing software on programmable devices | |
| Abran et al. | An analysis of the McCabe Cyclomatic complexity number | |
| US20200249916A1 (en) | Automated generation of software bindings | |
| Jiang et al. | A Large-scale Benchmark for Log Parsing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1176710 Country of ref document: HK |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: GR Ref document number: 1176710 Country of ref document: HK |