CN102081661A - Data integration method and system of heterogeneous relational database based on XML (Extensive Makeup Language) - Google Patents
Data integration method and system of heterogeneous relational database based on XML (Extensive Makeup Language) Download PDFInfo
- Publication number
- CN102081661A CN102081661A CN 201110021096 CN201110021096A CN102081661A CN 102081661 A CN102081661 A CN 102081661A CN 201110021096 CN201110021096 CN 201110021096 CN 201110021096 A CN201110021096 A CN 201110021096A CN 102081661 A CN102081661 A CN 102081661A
- Authority
- CN
- China
- Prior art keywords
- data
- integration
- query
- database
- xml
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000010354 integration Effects 0.000 title claims abstract description 88
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000002688 persistence Effects 0.000 claims abstract description 35
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 10
- 238000002360 preparation method Methods 0.000 claims abstract description 6
- 230000002085 persistent effect Effects 0.000 claims abstract 4
- 238000013507 mapping Methods 0.000 claims description 13
- 238000013461 design Methods 0.000 claims description 6
- 230000008676 import Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 3
- 238000012790 confirmation Methods 0.000 claims 1
- 238000013075 data extraction Methods 0.000 claims 1
- 238000012552 review Methods 0.000 claims 1
- 239000000047 product Substances 0.000 description 20
- 239000000284 extract Substances 0.000 description 9
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000004321 preservation Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000013501 data transformation Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000002620 method output Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000002045 lasting effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种基于XML的异构关系型数据库的数据集成方法和系统，旨在解决大量数据信息不能被高效地利用的问题。该方法包括如下步骤：获取数据集成需求和集成前期准备；若集成应用涉及了新数据库产品，则执行向系统中添加新支持的数据库产品；添加与集成有关的数据源；生成查询请求配置文档；在系统的持久化参数设置界面上录入持久化参数；在系统的执行集成计划界面上，选取集成计划文档，执行该集成计划，等待软件系统处理，如此反复执行步骤4到步骤6，直到完成所有的集成任务。基于XML的异构关系型数据库的数据集成系统是由集成任务管理器、数据源管理器、查询分解优化器、数据提取器、结果整合器和结果持久器所构成的功能模块构架。
The invention discloses a data integration method and system of an XML-based heterogeneous relational database, aiming at solving the problem that a large amount of data information cannot be efficiently utilized. The method includes the following steps: obtaining data integration requirements and pre-integration preparations; if the integrated application involves a new database product, adding a new supported database product to the system; adding a data source related to the integration; generating a query request configuration document; Enter the persistence parameters on the system’s persistent parameter setting interface; on the system’s execution integration plan interface, select the integration plan document, execute the integration plan, wait for the software system to process, and repeat steps 4 to 6 until all are completed integration tasks. The data integration system of XML-based heterogeneous relational database is a functional module framework composed of integration task manager, data source manager, query decomposition optimizer, data extractor, result integrator and result persister.
Description
Technical field
The present invention relates to a kind of data integrating method of areas of information technology, or rather, the present invention relates to a kind of data integrating method of the isomery relevant database based on XML, also we can say, the present invention relates to the data integrated system of a kind of enforcement based on the isomery relevant database of XML.
Background technology
In recent years, numerous enterprises is accompanied by professional development and has accumulated lot of data.Yet distributivity, isomerism and independence owing to Database Systems between enterprises and the enterprise cause the formation of " information island ", make mass data correctly not utilized efficiently.Therefore, solve this realistic problem, to the operation of enterprise with develop significant.
Specific industry such as bank, telecommunications etc. have had the ripe exchanges data and the Method and kit for of data integration at present.Also there is the demand of data integration in medium and small sized enterprises, but do not have unified industry standard, and data circulation form is mixed and disorderly, and the data integration difficulty needs a kind of extendability technical scheme strong, that versatility is good to solve this practical problems.
Traditional solution has two kinds:
1. build the central data warehouse, integrated data all are pooled to wherein.This method needs the soft hardware equipment of lot of data storage and maintenance, the expense costliness, and easy care not, and can not keep synchronously with source database, data user rate is not high.
2. federative database will be set up the interface of visit each other between all databases in the system.The maintenance and expansion meeting of this method relates to total system, and workload is big, and cost is very high.
Summary of the invention
Problem to be solved by this invention is to have overcome the technical matters that prior art exists, a kind of data integrating method based on the isomery relevant database of XML that is suitable for medium and small sized enterprises, Cheap highly effective, extendability and portable strong, cross operating system and database platform is provided, also we can say, the invention provides a kind of data integrated system of the isomery relevant database based on XML.
For solving the problems of the technologies described above, the present invention adopts following technical scheme to realize: the data integrating method of described isomery relevant database based on XML comprises the steps:
1. obtain data integration demand and integrated early-stage preparations.
2. if integrated application has related to the new database product, then carry out and add the new database product of supporting.
3. on the vision area of the data source of global query request configuration interface, add the destination data source of intending integrated data source and intending importing data.
4. ask on the configuration interface information of typing scheme, the preservation position of selection query requests document, generated query request configuration documentation in global query.
5. be provided with on the interface in the persistence parameter, typing persistence parameter, select the storing path of integrated planning documentation, generate integrated planning documentation, if the destination data table does not exist, in the persistence parameter information of the newly-built table of typing on the interface is set, creates a new tables of data, and then typing persistence parameter.
6. carrying out on the integration scheme interface, choose an integration scheme document, carry out this integration scheme, the wait software systems are handled, and carry out the state and the abnormal conditions that can show integrated planning execution on the integration scheme interface, after being finished, the result reports that the interface can show integrated result data statistical report form, data in the special processing exception table are carried out the 4th to 6 step so repeatedly, up to finishing all integration servers.
The step of obtaining data integration demand and integrated early-stage preparations described in the technical scheme is as follows:
1. obtain the data integration demand, determine integrated scope, formulate the data integration plan, write requirement specification.
2. determine the database environment at place, integrated data source, obtain the information of data source, comprising: the product type of database, version, database-name, database IP address, database service port and have login user name and password by JDBC manipulation data storehouse authority.
3. determine the integrated tables of data that relates to, obtain tables of data information, comprising: table schema, table name claim, the tabulation of list of fields and corresponding data type.
4. analyze demands, integration servers is resolved into some separate subtasks, determine the integrated logic of each subtask, integrated logical form is turned to query scheme, query scheme can be represented with Structured Query Language (SQL), should be noted that: each table and attribute column in the query statement all will have the sign of its affiliated database as prefix.
5. finish design to integrated destination data storehouse.
6. obtain persistence parameter as a result:
1) destination data source information.
2) destination data table information.
3) mapping relations of Query Result attribute and purpose Table Properties.
4) specify data to import exception table, note the information of this tables of data.
7. acquired parameters is verified and confirmed, finish the query scheme that obtained and the evaluation and the affirmation of destination data storehouse design proposal.
Described in the technical scheme if integrated application has related to the new database product, the step of then carry out adding the new database product of supporting is as follows:
1. obtain the information of all data types of the title that comprises data type in the newly-increased database product and characteristic.
2. according to the characteristic of data type, specify it to be mapped as a kind of conventional data type.
3. with adding in the data type dictionary of getting access to conventional data type map information and characteristic information.
A kind of data integrated system of data integrating method of the isomery relevant database based on XML, it comprises:
A global query's request and the persistence parameter that the collection user submits to is carried out validity checking to global query's request and persistence parameter, and the distribution integration servers is monitored integrating process, reports integrated result's integration servers manager.
Realized the data source dynamic management for one, the mapping of finish relation table is handled, and the data source manager of data source global view is provided for the user.
One is decomposed into plurality of sub query requests and global query's metadata with query requests according to the mapping relations of data, and the antithetical phrase query requests is optimized, and generates the query decomposition optimizer of subquery plan.
Give the data extract engine with the subquery planned assignment that receives for one, bottom data source issue SQL query is extracted data, and query results is converted to the data extractor of intermediate result XML data.
One receives intermediate result XML data and global query's metadata, utilizes the relational algebra engine, and middle result is integrated, and generates the final result's of global query Query Result integrator.
Result data after will integrating imports to the persistor as a result in the data with existing storehouse.
Compared with prior art the invention has the beneficial effects as follows:
1. the data integrating method of the isomery relevant database based on XML of the present invention can guarantee data source not to be made any change in the process of data integration, and therefore integrated risk and cost are little.
2. the data integrating method of the isomery relevant database based on XML of the present invention has used dynamic data source management, and inquiry gained data are latest data, have guaranteed the consistance of data.
3. the data integrating method of the isomery relevant database based on XML of the present invention need not to build the central data warehouse, has saved the expense of database software product and mass memory unit, has reduced the risk of integrated project.
4. the data integrating method of the isomery relevant database based on XML of the present invention is by the java applet language compilation with professional platform independence, and system has platform independence, transplants easily.
5. when the data integrating method of the isomery relevant database based on XML of the present invention need be supported new database product or data source, only need register corresponding project in data source dictionary or data type dictionary gets final product, this system that makes has good extendability, safeguard that easily cost is lower.
6. the data integrating method of the isomery relevant database based on XML of the present invention needs the data extracted by integrated demand decision, has carried out query optimization before data extract, and data user rate is very high.
Description of drawings
The present invention is further illustrated below in conjunction with accompanying drawing:
Fig. 1 is the flow chart of steps of the data integrating method of the isomery relevant database based on XML of the present invention;
Fig. 2 is the functional module construction block diagram of the data integrated system of the isomery relevant database based on XML of the present invention;
Fig. 3 is the functional module construction block diagram of data source manager of the data integrated system of the isomery relevant database based on XML of the present invention;
Fig. 4～Fig. 6 is a kind of method for expressing that utilizes the data integrated system query requests of the isomery relevant database based on XML of the present invention;
Fig. 7 is the user's visualization interface of a kind of global query configuration that utilizes the data integrated system of the isomery relevant database based on XML of the present invention;
Fig. 8 is a kind of intermediate result XML data representation of data integrated system method of utilizing the isomery relevant database based on XML of the present invention.
Embodiment
Below in conjunction with accompanying drawing the present invention is explained in detail:
The invention provides a kind of data integrating method that is suitable for medium and small sized enterprises, Cheap highly effective, extendability and portable strong, cross operating system and database platform based on the isomery relevant database of XML.
This method is based on a self-editing computer program based on the data integration of the isomery relevant database of XML, this computer program is to operate in the network environment, based on the Java of autonomous definition and XML data type system, have dynamic data source control, integration servers configuration and optimize, data extract and integration, the software systems of the cross-platform data integration of persistence function as a result.This method has provided the solution of the newly-increased database products of software systems simultaneously, and the treatment step of concrete data integration application.
I. based on the data integrating method of the isomery relevant database of XML
According to the step of computer program means flow process, as follows based on the step of the data integrating method of the isomery relevant database of XML:
1. obtain data integration demand and integrated early-stage preparations
1) obtains the data integration demand, determine integrated scope, formulate the data integration plan, write requirement specification.
2) determine the database environment at place, integrated data source.Obtain the information of data source, comprising: the product type of database, version, database-name, database IP address, the database service port, and have login user name and password by JDBC manipulation data storehouse authority.
3) determine the integrated tables of data that relates to, obtain tables of data information, comprising: table schema, table name claim, the tabulation of list of fields and corresponding data type.
4) analyze demands resolves into some separate subtasks with integration servers, determines the integrated logic of each subtask, and integrated logical form is turned to query scheme.Query scheme can be represented with Structured Query Language (SQL).Should be noted that: each table and attribute column in the query statement all will have the sign of its affiliated database as prefix.
5) finish design to integrated destination data storehouse.
6) obtain persistence parameter as a result:
(1) destination data source information.
(2) destination data table information.
(3) mapping relations of the attribute of Query Result attribute and purpose table.
(4) specify data to import exception table, note the information of this tables of data.
7) acquired parameters is verified and confirmed.Finish the query scheme that obtained and the evaluation and the affirmation of destination data storehouse design proposal.
2. if integrated application has related to the new database product, then carry out and in software systems, add the new database product of supporting.
With the vision area of the data source of the global query of system of integrated software request configuration interface on, add and integrated relevant data source, comprise the destination data source of intending integrated data source and intending the importing data.
4. choose a query scheme, on the interface of global query's request configuration, the information of this scheme of typing, the preservation position of selection query requests document generates a query requests configuration documentation.
5. the persistence parameter in system is provided with on the interface, and typing persistence parameter is selected the storing path of integrated planning documentation, generates an integration scheme document.If the destination data table do not exist, can be on the interface information of the newly-built table of typing, create a new tables of data, and then typing persistence parameter.
6. on the execution integration scheme interface of system, choose an integration scheme document, carry out this integration scheme, wait for the software systems processing, can show the state and the abnormal conditions of integrated planning execution on the interface.After being finished, the result reports that the interface can show integrated result data statistical report form, the data in the special processing exception table.Execution in step 4 is to step 6, up to finishing all integration servers so repeatedly.
II. it is as follows to add the step of the new database product of supporting based on the data integrating method of the isomery relevant database of XML in software systems:
1. obtain the information of all data types in the newly-increased database product, comprise the title and the characteristic of data type.
2. according to the characteristic of data type, specify it to be mapped as a kind of conventional data type.
3. with adding in the data type dictionary of system of integrated software of getting access to conventional data type map information and characteristic information.
III. based on the data integrated system of the isomery relevant database of XML
Consult Fig. 2, setting forth according to device for same computer program, is by integration servers manager, data source manager, query decomposition optimizer, data extractor, integrator and persistor constituted as a result functional module construction as a result based on the data integrated system of the isomery relevant database of XML.
1. integration servers manager
Collect global query's request and persistence parameter that the user submits to, validity checking is carried out in global query's request and persistence parameter, the distribution integration servers, the monitoring integrating process is reported integrated result.The implementation procedure of integration servers manager is:
1) determine the representation of query requests, this expression must have following characteristics:
(1) accurately giving expression to each relates to integrated tables of data unambiguity;
(2) can know the semanteme that gives expression to Structured Query Language (SQL).Fig. 4～Fig. 6 has provided a kind of embodiment of method for expressing.
2) create user's visualization interface, comprise four parts: global query's request configuration, persistence parameter are provided with, carry out integration scheme and result's report.
Must there be following characteristics at global query requesting users interface:
(1) shows the unified view of data source table schema;
(2) can give expression to the semanteme of Structured Query Language (SQL);
(3) provide necessary information for reference and that select, user-friendly.
Consult Fig. 7, provided the solution of a kind of global query request configuration interface among the figure, the left side is the data source vision area, shows available data sources, can add on this vision area, removes and the refresh data source.The right is the query configuration vision area, is divided into six subregions, the clause of the corresponding structuralized query of each subregion.Can add some projects in each subregion, comprising in the project can be for the tables of data of user's selection and the territory of column information, sign of operation and confession user input.The bottom is a vision area as a result, the preservation position that can create query requests and configuration querying request.
The persistence parameter is provided with the parameter that interfacial energy is collected the Query Result persistence, comprising: the mapping relations of destination data source information, destination data table information, Query Result attribute and purpose Table Properties, data importing exception table information.Need to select a query requests before collecting the persistence parameter, the Table Properties of persistence parameter should be corresponding consistent on number and type with the attribute of Query Result.On this interface, can dispose the preservation position of integration scheme.In addition, can pass through this interface newdata table in data source.
Carry out the integration scheme interface, embody following function: Integrated Solution is selected, integration scheme executing state feedback, mistake and abnormal prompt.
The situation information of integration scheme execution that the result has reported interface display shows integrated result data statistical report form.
3) make up and data source manager, query decomposition optimizer, data extractor, the integrator and the communication interface of persistor as a result as a result.Determine semantic and wrong and the abnormity processing mode of message and data transfer.
4) realize business processing flow:
(1) global query's request configuration interface and user interactions are communicated by letter with the data source manager, can finish the interpolation of data source, remove refresh function.
(2) global query's request configuration interface and user interactions are communicated by letter with the data source manager, obtain global query's information, generate global query's request configuration documentation, and it is carried out morphology, the checking inspection of grammer, if find mistake, and the feedback user corrigendum.
(3) the persistence parameter is provided with interface and user interactions, communicates by letter with the data source manager, obtains query requests configuration documentation and persistence parameter, generates integrated planning documentation.If the destination data table does not still exist, can use to build and show to specify the persistence parameter again after the device newdata table.Build the treatment scheme of table device: read the user and build the table parameter; Be converted into the DDL code; Mutual with the data source manager data, obtain data source information; Set up database and connect, carry out and build table code; Catch abnormal information, the table result is built in report.
(4) user and execution integration scheme interface alternation obtain the integration scheme document.After integrated order is sent, obtain global query's request, and pass to the query decomposition optimizer.Set up and the query decomposition optimizer, data extractor, communicating to connect of integrator monitored executing state as a result, collects unexpected message and statistical information, writes daily record, to the user report implementation status.Read the persistence parameter, the result data of persistence parameter and integration is passed to the lasting data device, and monitoring persistence state, collect unusual and statistical information, write daily record, to user report persistence situation.
(5) after persistence is finished, collect the integrating process statistical information, generate and show integrated result data statistical report form.
2. data source manager
Consult Fig. 3, dynamic data source management is provided, the mapping of finish relation table is handled, for the user provides the data source global view.Represented the internal structure of data source manager among the figure:
1) set up the data source dictionary, it has preserved the data source that data integration relates to and the information of tables of data thereof.Wherein the information of data source comprises: the product type of database, and version, database-name, database IP address, the database service port, and have login username and password by JDBC manipulation data storehouse authority.Tables of data information comprises: table schema, table name claim, the tabulation of field and corresponding data type.The data source manager is safeguarded this data source dictionary in system's operational process.The interpolation of data source, remove the information updating of the data source dictionary with the refresh operation correspondence.
2) create four kinds of conventional data types, CHAR, NUMBER, DATE and BOOLEAN, every kind of data type has the metamessage of this kind data type, and it comprises: the characteristic of conventional data type, with the compatibility of other conventional data types and the condition of compatible conversion.
3) create the data type dictionary, it has preserved the map information of database data type and conventional data type.When system adds new data source product, need in the data type dictionary, add corresponding map entries.
4) make up the global view of data source, it has showed the data source of registering in the data source dictionary, tables of data, and the information of attribute column also comprises the conventional data type of the data type correspondence of each attribute column.Every data source of registering in the data source dictionary, tables of data, attribute column all are visible in global view.
5) make up metamessage and extract engine, this engine comprises the common interface of data access and the general extracting method of metamessage.Metamessage extracts the operational order that engine is accepted the user, and data source information and table name information are set up JDBC and connected, the availability in verification msg source, and from data source, read the information of attribute column and corresponding data type, information data is synchronized in the data source dictionary.
3. query decomposition optimizer
The effect of query decomposition optimizer is that query requests is decomposed into plurality of sub query requests and global query's metadata according to the mapping relations of data, and the antithetical phrase query requests is optimized, and generates the subquery plan.The course of work of query decomposition optimizer is as follows:
1) receives query requests, generate the relational algebra expression tree.Table among the From clause is as the leafy node of relational algebra expression tree.Utilize Where clause's condition of contact that above-mentioned leafy node is carried out merger, the wherein preferential merger of the condition of contact of the table of same database among the Where clause, generate the binary tree forest, with the above-mentioned binary tree forest of the condition of contact merger of disparate databases table, generate a binary tree then.When binary tree of the not enough generation of condition of contact, with the remaining binary tree forest of cartesian product merger.Projection, gathering and common alternative condition all are placed on tree root, wait to be optimized.
2) under selecting, push away, push away under the projection, assemble the principle abbreviation relational algebra expression tree that pushes away down.
3) generated query plan and query metadata.Push away after the optimization down through various, travel through the node in the relational algebra expression tree, find the node of this condition: be that relational algebra in the subtree of root node only relates to same database with this node, and its father node does not possess this character.To be the relational algebra of the subtree expression of root with these nodes, be converted to the SQL query plan.Among the Where clause, relate to the condition that the multiple database table connects, can generate global query's metadata.
4) with resulting plurality of sub inquiry plan, send to data extractor.
5) with resulting global query metadata, send to integrator as a result.
6) with job schedule, unusual or mistake sends to the integration servers manager.
4. data extractor
Mainly acting as of data extractor: receive the subquery plan, and they are assigned to the data extract engine, each data extract engine will be carried out a sub-inquiry plan, bottom data source issue SQL query be extracted data, and query results is converted to intermediate result XML data.The data extractor implementation procedure is as follows:
1) determine intermediate result XML data representation format, it has following characteristics:
(1) gives expression to data list structure information;
(2) give expression to data recording information;
(3) give expression to the scale information of tables of data;
(4) clear in structure is simple, unambiguity, and redundant data is few, resolves easily and generates.Intermediate result XML data definition can be become list structure and two XML document of table data, an XML list structure of all unique reference of each attribute column document of XML table data file.
Consult Fig. 8, provided a kind of implementation among the figure: the table data file has write down the information of each attribute column of the data scale of this data file, every record.Wherein each attribute column information comprises list structure document title, column number and the data of reference.The list structure document has write down conventional data type and the type attribute thereof that numbering, title, data type and its mapping of information, each attribute column of title, the data source of this list structure document become.
2) virtual file storage is set up super large file management mechanism.Can rewrite the File class of Java language, making files classes logically is a file, and physically is a file group.File in the file group prevents that less than the size of the upper limit of file system file from overflowing.
3) set up memory buffer mechanism, adopt output intent to generate XML document based on stream.
4) make up the data extract engine, this engine comprises the common interface of data access and the general extracting method of data.
5) realize the data extract flow process:
(1) receives the subquery plan.
(2) communicate by letter with the data source manager data and obtain the metamessage of subquery.
(3) set up JDBC and connect, obtain the result set of subquery, the result set data are converted to intermediate result XML data.
(4) give integrator as a result with intermediate result XML data transfer.
5. integrator as a result
Integrator receives intermediate result XML data and global query's metadata as a result, utilizes the relational algebra engine, and middle result is integrated, and generates the final result of global query.The integrator implementation procedure is as follows as a result:
1) makes up the ordering engine
Be input as an XML table data file, the numbering of the XML list structure of quasi-ordering and key attribute row, the conventional data type of attribute column, lifting order parameter.Be output as orderly intermediate result XML data, implementation procedure is as follows:
(1) in internal memory, sets up the data structure of data recording.
(2) according to the configuration of system, the upper limit is held in the definition internal sort.
(3) set up memory buffer mechanism, select STAX instrument analyzing XML file for use.Employing generates XML document based on the output intent of stream.
(4) read in data in bulk, carry out the merger internal sort, output to the external memory file.
(5) do the merger external sort, obtain ranking results.
2) make up nature and connect engine
Be input as two XML table data files, do the connection attribute column number, tag align sort.Be output as the intermediate result XML data after table connects.Implementation procedure is as follows:
(1) if XML table data do not sort according to the connection attribute row, calls the ordering engine and list ordering in connection;
(2) in internal memory, set up the data structure of data recording;
(3) resolve two XML table data files according to the order of sequence with STAX, run into the identical attribute column of value, just generate a record, use method output based on stream.Finish up to the document parsing, obtain connecting the result.
3) make up the cartesian product engine
Be input as two XML table data files doing cartesian product.Be output as the intermediate result XML data after the cartesian product computing.Implementation procedure is as follows:
(1) in internal memory, sets up the data structure of data recording.
(2) resolve one of them XML table data file with STAX,, travel through each the bar record Rj in another XML table data file for each bar record Ri, (Ri Rj), uses the method output based on stream to the new record that generates, finish up to the document parsing, obtain connecting the result.
(3) the table data file is resolved and is finished, and obtains the cartesian product result.
4) make up the gathering engine
Be input as the table data file of doing gathering, groupby clause, having clause.Aggregation operator type and attribute column.Implementation procedure is as follows:
(1) in internal memory, sets up the data structure of data recording.
(2) according to system configuration, the definition internal memory holds the record upper limit.
(3) call the ordering engine according to the attribute column among the groupby clause.
(4) with STAX resolution table data file, sweep record according to the order of sequence.The packet attributes row arrive new value whenever and generate a grouping, do screening with having clause's condition.The record of grouping sum surpasses internal memory and holds the record upper limit after screening, and whole records of grouping export external memory in the mode of stream, continues to handle next bar record, until the end of scan.A scanning is carried out in group record, done aggregate operation, obtain a record, the output result.The group record been scanned obtains assembling the result.
5) realize the data integration flow process
(1) obtains query metadata, call in to the data structure of internal memory.
(2) determine to connect the select progressively strategy of showing, can utilize heuristic rule: select two less tables of data scale sum to make table at every turn and connect.
(3) according to the metadata of Query Result, the definite relational algebra operation that will do, call relation algebraic operation engine is handled and is integrated middle result.
(4) with job schedule, unusual or mistake sends to the integration servers manager.The integral data result is passed to the integration servers manager.
6. persistor as a result
Result data after integrating is imported in the data with existing storehouse.The implementation procedure of persistor is as follows as a result:
1) makes up data transformation engine.Every record of its traversal queries result according to the Query Result attribute column with import corresponding relation between the Table Properties row, generates SQL and inserts the statement script.
2) data importing engine, this engine comprise the common interface and the universal method of data importing.It reads persistence information and data source information, sets up JDBC and connects, and carries out SQL and inserts the statement script, catches unusual in the importing process, inserts unusual data trial and is inserted in the exception table.
3) realize the data persistence flow process.Obtain the result data of persistence parameter and integration, call data transformation engine result data is converted to the SQL script, call the data importing engine, carry out script in batches.With the state of data importing with send to the integration servers manager unusually.
Claims (4)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110021096 CN102081661A (en) | 2011-01-19 | 2011-01-19 | Data integration method and system of heterogeneous relational database based on XML (Extensive Makeup Language) |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110021096 CN102081661A (en) | 2011-01-19 | 2011-01-19 | Data integration method and system of heterogeneous relational database based on XML (Extensive Makeup Language) |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102081661A true CN102081661A (en) | 2011-06-01 |
Family
ID=44087624
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110021096 Pending CN102081661A (en) | 2011-01-19 | 2011-01-19 | Data integration method and system of heterogeneous relational database based on XML (Extensive Makeup Language) |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102081661A (en) |
Cited By (42)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521254A (en) * | 2011-11-17 | 2012-06-27 | 广东电网公司电力科学研究院 | Uniform access method of isomeric database |
| CN102521711A (en) * | 2011-12-26 | 2012-06-27 | 北京瑞风协同科技股份有限公司 | Control method and device for construction of quality characteristic data model of equipment |
| CN102750358A (en) * | 2012-06-12 | 2012-10-24 | 中国电力科学研究院 | Mapping method and system of system data model to common information model (CIM) |
| CN102789491A (en) * | 2012-07-03 | 2012-11-21 | 河海大学 | Configurable data subscribing and publishing system and method thereof |
| CN103294821A (en) * | 2013-06-17 | 2013-09-11 | 北京工业大学 | XML data query result visiting method based on multi-level subquery result branch trees |
| CN103309977A (en) * | 2013-06-14 | 2013-09-18 | 广东电网公司电力科学研究院 | Heterogeneous data resource integration method |
| CN103336843A (en) * | 2013-07-18 | 2013-10-02 | 山东中创软件工程股份有限公司 | Data integration method and device |
| CN103368980A (en) * | 2012-03-26 | 2013-10-23 | 深圳市财付通科技有限公司 | Method and device for JSONP data request |
| CN103455381A (en) * | 2012-05-29 | 2013-12-18 | 国际商业机器公司 | Method and system for de-serializing source object of source software into target software component |
| CN103793399A (en) * | 2012-10-31 | 2014-05-14 | 北京航天长峰科技工业集团有限公司 | Method for integrating information resources of system of politics and law |
| CN103902671A (en) * | 2014-03-19 | 2014-07-02 | 北京科技大学 | Dynamic integration method and system of multi-source heterogeneous data |
| CN103942234A (en) * | 2013-01-21 | 2014-07-23 | 中国电信股份有限公司 | Method for operating multiple heterogeneous databases, middleware device and system |
| CN104346377A (en) * | 2013-07-31 | 2015-02-11 | 克拉玛依红有软件有限责任公司 | Method for integrating and exchanging data on basis of unique identification |
| CN105243162A (en) * | 2015-10-30 | 2016-01-13 | 方正国际软件有限公司 | Relational database storage-based objective data model query method and device |
| CN105359141A (en) * | 2013-05-17 | 2016-02-24 | 甲骨文国际公司 | Supporting combination of flow based ETL and entity relationship based ETL |
| CN105488229A (en) * | 2016-01-20 | 2016-04-13 | 航天科工智慧产业发展有限公司 | Data exchange and integration method applied to heterogeneous data environment |
| CN105956126A (en) * | 2016-05-06 | 2016-09-21 | 南京国电南自电网自动化有限公司 | XML (X Exrensible Markup Language) query method based on primary and secondary classification of keywords |
| CN106815371A (en) * | 2017-02-06 | 2017-06-09 | 浪潮通用软件有限公司 | A kind of method for reading data realized by visual configuration across data source |
| CN106844485A (en) * | 2016-12-23 | 2017-06-13 | 航天星图科技（北京）有限公司 | A kind of system and method for enterprise's heterogeneous database intelligent integrated |
| CN107368588A (en) * | 2017-07-24 | 2017-11-21 | 人教数字出版有限公司 | A kind of heterogeneous resource Homogeneous method and device |
| CN107480225A (en) * | 2017-09-11 | 2017-12-15 | 爱普（福建）科技有限公司 | Realize the method and computer program product of control station and third party database data sharing |
| CN107679071A (en) * | 2017-08-22 | 2018-02-09 | 中国科学院计算机网络信息中心 | A kind of generic data service of facing relation database customizes method for packing |
| CN108132936A (en) * | 2016-11-30 | 2018-06-08 | 北京国双科技有限公司 | Data lead-in method and device |
| CN108235755A (en) * | 2015-07-15 | 2018-06-29 | 魏庆军 | A kind of method and system of internet of things net controller user interface |
| CN109067558A (en) * | 2018-06-11 | 2018-12-21 | 玖富金科控股集团有限责任公司 | data service method and system |
| CN109145025A (en) * | 2018-09-14 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of data query method, apparatus and service server that multi-data source is integrated |
| CN109344166A (en) * | 2018-08-14 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Monitoring method, computer readable storage medium and the terminal device of database |
| CN109495581A (en) * | 2018-12-13 | 2019-03-19 | 爱普（福建）科技有限公司 | A kind of communication means and device of polyisocyanate structure control station |
| CN109783694A (en) * | 2019-01-30 | 2019-05-21 | 清华大学 | XML-based cross-platform large-scale instrument sharing information integration method and system |
| CN110069559A (en) * | 2019-03-21 | 2019-07-30 | 中国人民解放军陆军工程大学 | Heterogeneous information system data analysis and integration method with high automatic control |
| CN110544092A (en) * | 2019-08-22 | 2019-12-06 | 杭州趣链科技有限公司 | Dynamic newly-added multi-type database data operation chaining method for block chain |
| CN110597844A (en) * | 2019-08-14 | 2019-12-20 | 中国平安财产保险股份有限公司 | Heterogeneous database data unified access method and related equipment |
| CN110781189A (en) * | 2019-10-25 | 2020-02-11 | 北京达佳互联信息技术有限公司 | Document platform construction method and device, electronic equipment and storage medium |
| CN110909059A (en) * | 2019-11-25 | 2020-03-24 | 杭州晨鹰军泰科技有限公司 | Data integration system, method, equipment and storage medium |
| CN111177134A (en) * | 2019-12-26 | 2020-05-19 | 上海科技发展有限公司 | Data quality analysis method, device, terminal and medium suitable for mass data |
| CN111221791A (en) * | 2018-11-27 | 2020-06-02 | 中云开源数据技术（上海）有限公司 | Method for importing multi-source heterogeneous data into data lake |
| CN111443970A (en) * | 2020-03-24 | 2020-07-24 | 山东浪潮通软信息科技有限公司 | Method, device and equipment for assembling multi-source data and readable medium |
| CN112817580A (en) * | 2021-01-27 | 2021-05-18 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113806327A (en) * | 2020-06-16 | 2021-12-17 | 华为技术有限公司 | Database design method and device and related equipment |
| CN114217899A (en) * | 2021-12-15 | 2022-03-22 | 平安国际智慧城市科技股份有限公司 | Data persistence method and device, electronic equipment and storage medium |
| CN114443699A (en) * | 2022-01-27 | 2022-05-06 | 腾讯科技（深圳）有限公司 | Information query method, apparatus, computer equipment, and computer-readable storage medium |
| CN116932575A (en) * | 2023-09-12 | 2023-10-24 | 长城证券股份有限公司 | Spark-based cross-data source operation method, device and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101067814A (en) * | 2007-05-10 | 2007-11-07 | 浪潮集团山东通用软件有限公司 | Mapping conversion method between data access level Xml format data and relational data |
| CN101739436A (en) * | 2009-09-28 | 2010-06-16 | 孙彬 | XML-based flexible data migration method |
-
2011
- 2011-01-19 CN CN 201110021096 patent/CN102081661A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101067814A (en) * | 2007-05-10 | 2007-11-07 | 浪潮集团山东通用软件有限公司 | Mapping conversion method between data access level Xml format data and relational data |
| CN101739436A (en) * | 2009-09-28 | 2010-06-16 | 孙彬 | XML-based flexible data migration method |
Cited By (61)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521254A (en) * | 2011-11-17 | 2012-06-27 | 广东电网公司电力科学研究院 | Uniform access method of isomeric database |
| CN102521711A (en) * | 2011-12-26 | 2012-06-27 | 北京瑞风协同科技股份有限公司 | Control method and device for construction of quality characteristic data model of equipment |
| CN102521711B (en) * | 2011-12-26 | 2016-03-02 | 北京瑞风协同科技股份有限公司 | The control method of the data model structure of equipment quality characteristic and device |
| CN103368980A (en) * | 2012-03-26 | 2013-10-23 | 深圳市财付通科技有限公司 | Method and device for JSONP data request |
| CN103455381A (en) * | 2012-05-29 | 2013-12-18 | 国际商业机器公司 | Method and system for de-serializing source object of source software into target software component |
| CN102750358A (en) * | 2012-06-12 | 2012-10-24 | 中国电力科学研究院 | Mapping method and system of system data model to common information model (CIM) |
| CN102789491A (en) * | 2012-07-03 | 2012-11-21 | 河海大学 | Configurable data subscribing and publishing system and method thereof |
| CN102789491B (en) * | 2012-07-03 | 2016-03-16 | 河海大学 | A kind of configurable data subscription and delivery system and method thereof |
| CN103793399A (en) * | 2012-10-31 | 2014-05-14 | 北京航天长峰科技工业集团有限公司 | Method for integrating information resources of system of politics and law |
| CN103942234A (en) * | 2013-01-21 | 2014-07-23 | 中国电信股份有限公司 | Method for operating multiple heterogeneous databases, middleware device and system |
| CN105359141A (en) * | 2013-05-17 | 2016-02-24 | 甲骨文国际公司 | Supporting combination of flow based ETL and entity relationship based ETL |
| CN103309977A (en) * | 2013-06-14 | 2013-09-18 | 广东电网公司电力科学研究院 | Heterogeneous data resource integration method |
| CN103294821B (en) * | 2013-06-17 | 2016-01-20 | 北京工业大学 | Based on the XML data query result access method of multilayer subquery results branch tree |
| CN103294821A (en) * | 2013-06-17 | 2013-09-11 | 北京工业大学 | XML data query result visiting method based on multi-level subquery result branch trees |
| CN103336843A (en) * | 2013-07-18 | 2013-10-02 | 山东中创软件工程股份有限公司 | Data integration method and device |
| CN103336843B (en) * | 2013-07-18 | 2017-02-15 | 山东中创软件工程股份有限公司 | Data integration method and device |
| CN104346377A (en) * | 2013-07-31 | 2015-02-11 | 克拉玛依红有软件有限责任公司 | Method for integrating and exchanging data on basis of unique identification |
| CN104346377B (en) * | 2013-07-31 | 2017-08-08 | 克拉玛依红有软件有限责任公司 | A kind of data integration and transfer method based on unique mark |
| CN103902671A (en) * | 2014-03-19 | 2014-07-02 | 北京科技大学 | Dynamic integration method and system of multi-source heterogeneous data |
| CN103902671B (en) * | 2014-03-19 | 2018-04-13 | 北京科技大学 | A kind of dynamic integrity method and system of isomerous multi-source data |
| CN108235755B (en) * | 2015-07-15 | 2021-07-20 | 魏庆军 | Method and system for user interface of IoT controller |
| CN108235755A (en) * | 2015-07-15 | 2018-06-29 | 魏庆军 | A kind of method and system of internet of things net controller user interface |
| CN105243162B (en) * | 2015-10-30 | 2018-10-30 | 方正国际软件有限公司 | Objectification data model querying method and device based on relational data library storage |
| CN105243162A (en) * | 2015-10-30 | 2016-01-13 | 方正国际软件有限公司 | Relational database storage-based objective data model query method and device |
| CN105488229A (en) * | 2016-01-20 | 2016-04-13 | 航天科工智慧产业发展有限公司 | Data exchange and integration method applied to heterogeneous data environment |
| CN105956126A (en) * | 2016-05-06 | 2016-09-21 | 南京国电南自电网自动化有限公司 | XML (X Exrensible Markup Language) query method based on primary and secondary classification of keywords |
| CN108132936A (en) * | 2016-11-30 | 2018-06-08 | 北京国双科技有限公司 | Data lead-in method and device |
| CN106844485A (en) * | 2016-12-23 | 2017-06-13 | 航天星图科技（北京）有限公司 | A kind of system and method for enterprise's heterogeneous database intelligent integrated |
| CN106815371A (en) * | 2017-02-06 | 2017-06-09 | 浪潮通用软件有限公司 | A kind of method for reading data realized by visual configuration across data source |
| CN107368588B (en) * | 2017-07-24 | 2020-09-01 | 人教数字出版有限公司 | Heterogeneous resource isomorphism method and device |
| CN107368588A (en) * | 2017-07-24 | 2017-11-21 | 人教数字出版有限公司 | A kind of heterogeneous resource Homogeneous method and device |
| CN107679071A (en) * | 2017-08-22 | 2018-02-09 | 中国科学院计算机网络信息中心 | A kind of generic data service of facing relation database customizes method for packing |
| CN107679071B (en) * | 2017-08-22 | 2020-12-18 | 中国科学院计算机网络信息中心 | A custom packaging method for general data services for relational databases |
| CN107480225A (en) * | 2017-09-11 | 2017-12-15 | 爱普（福建）科技有限公司 | Realize the method and computer program product of control station and third party database data sharing |
| CN109067558A (en) * | 2018-06-11 | 2018-12-21 | 玖富金科控股集团有限责任公司 | data service method and system |
| CN109344166A (en) * | 2018-08-14 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Monitoring method, computer readable storage medium and the terminal device of database |
| CN109344166B (en) * | 2018-08-14 | 2023-06-09 | 中国平安人寿保险股份有限公司 | Database monitoring method, computer readable storage medium and terminal device |
| CN109145025A (en) * | 2018-09-14 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of data query method, apparatus and service server that multi-data source is integrated |
| CN109145025B (en) * | 2018-09-14 | 2021-09-24 | 创新先进技术有限公司 | Multi-data-source integrated data query method and device and service server |
| CN111221791A (en) * | 2018-11-27 | 2020-06-02 | 中云开源数据技术（上海）有限公司 | Method for importing multi-source heterogeneous data into data lake |
| CN109495581A (en) * | 2018-12-13 | 2019-03-19 | 爱普（福建）科技有限公司 | A kind of communication means and device of polyisocyanate structure control station |
| CN109783694A (en) * | 2019-01-30 | 2019-05-21 | 清华大学 | XML-based cross-platform large-scale instrument sharing information integration method and system |
| CN109783694B (en) * | 2019-01-30 | 2021-02-12 | 清华大学 | Cross-platform large instrument shared information integration method and system based on XML |
| CN110069559A (en) * | 2019-03-21 | 2019-07-30 | 中国人民解放军陆军工程大学 | Heterogeneous information system data analysis and integration method with high automatic control |
| CN110597844A (en) * | 2019-08-14 | 2019-12-20 | 中国平安财产保险股份有限公司 | Heterogeneous database data unified access method and related equipment |
| CN110597844B (en) * | 2019-08-14 | 2023-07-21 | 中国平安财产保险股份有限公司 | Unified access method for heterogeneous database data and related equipment |
| CN110544092B (en) * | 2019-08-22 | 2022-04-01 | 杭州趣链科技有限公司 | Dynamic newly-added multi-type database data operation chaining method for block chain |
| CN110544092A (en) * | 2019-08-22 | 2019-12-06 | 杭州趣链科技有限公司 | Dynamic newly-added multi-type database data operation chaining method for block chain |
| CN110781189A (en) * | 2019-10-25 | 2020-02-11 | 北京达佳互联信息技术有限公司 | Document platform construction method and device, electronic equipment and storage medium |
| CN110909059A (en) * | 2019-11-25 | 2020-03-24 | 杭州晨鹰军泰科技有限公司 | Data integration system, method, equipment and storage medium |
| CN111177134A (en) * | 2019-12-26 | 2020-05-19 | 上海科技发展有限公司 | Data quality analysis method, device, terminal and medium suitable for mass data |
| CN111443970A (en) * | 2020-03-24 | 2020-07-24 | 山东浪潮通软信息科技有限公司 | Method, device and equipment for assembling multi-source data and readable medium |
| CN111443970B (en) * | 2020-03-24 | 2023-11-03 | 浪潮通用软件有限公司 | Method, device, equipment and readable medium for assembling multi-source data |
| CN113806327A (en) * | 2020-06-16 | 2021-12-17 | 华为技术有限公司 | Database design method and device and related equipment |
| CN112817580A (en) * | 2021-01-27 | 2021-05-18 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112817580B (en) * | 2021-01-27 | 2023-09-01 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114217899A (en) * | 2021-12-15 | 2022-03-22 | 平安国际智慧城市科技股份有限公司 | Data persistence method and device, electronic equipment and storage medium |
| CN114217899B (en) * | 2021-12-15 | 2023-10-17 | 深圳平安智慧医健科技有限公司 | Data persistence method, device, electronic equipment and storage medium |
| CN114443699A (en) * | 2022-01-27 | 2022-05-06 | 腾讯科技（深圳）有限公司 | Information query method, apparatus, computer equipment, and computer-readable storage medium |
| CN116932575A (en) * | 2023-09-12 | 2023-10-24 | 长城证券股份有限公司 | Spark-based cross-data source operation method, device and storage medium |
| CN116932575B (en) * | 2023-09-12 | 2023-12-15 | 长城证券股份有限公司 | Spark-based cross-data source operation method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102081661A (en) | Data integration method and system of heterogeneous relational database based on XML (Extensive Makeup Language) | |
| US7917463B2 (en) | System and method for data warehousing and analytics on a distributed file system | |
| JP7074307B2 (en) | Multi-center medical data structure standardization system based on generic data model | |
| US7805341B2 (en) | Extraction, transformation and loading designer module of a computerized financial system | |
| US10127278B2 (en) | Processing database queries using format conversion | |
| US8359305B1 (en) | Query metadata engine | |
| EP2608074B1 (en) | Systems and methods for merging source records in accordance with survivorship rules | |
| CN107402988A (en) | A kind of distributed NewSQL Database Systems and Query semi-structured for data method | |
| CN105224631B (en) | The method built the system of the open cloud of industry and work out XBRL financial statement | |
| CN110134671B (en) | Traceability application-oriented block chain database data management system and method | |
| EP2784700A2 (en) | Integration of transactional and analytical capabilities of a database management system | |
| CN103942234A (en) | Method for operating multiple heterogeneous databases, middleware device and system | |
| US8122044B2 (en) | Generation of business intelligence entities from a dimensional model | |
| CN105138661A (en) | Hadoop-based k-means clustering analysis system and method of network security log | |
| CN107491561A (en) | A kind of urban transportation heterogeneous data integrated system and method based on body | |
| CN101984439A (en) | Method for realizing optimization of data source extensive makeup language (XML) query system based on sub-queries | |
| CN103262076A (en) | Analytical data processing | |
| CN109241054A (en) | A kind of multimodal data library system, implementation method and server | |
| CN107291471B (en) | Meta-model framework system supporting customizable data acquisition | |
| CN108009270A (en) | A kind of text searching method calculated based on distributed memory | |
| US8639717B2 (en) | Providing access to data with user defined table functions | |
| CN112735571A (en) | Medical health data uploading management platform | |
| CN101882147B (en) | Curve data storage device and method | |
| CN104834742A (en) | ETL architecture management method based on SCA | |
| US7574329B1 (en) | Object model for decision and issue tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20110601 |