CN1152334C - Autonomous intelligent isomeri data integration system and method - Google Patents
Autonomous intelligent isomeri data integration system and method Download PDFInfo
- Publication number
- CN1152334C CN1152334C CNB021489912A CN02148991A CN1152334C CN 1152334 C CN1152334 C CN 1152334C CN B021489912 A CNB021489912 A CN B021489912A CN 02148991 A CN02148991 A CN 02148991A CN 1152334 C CN1152334 C CN 1152334C
- Authority
- CN
- China
- Prior art keywords
- data
- network
- engine
- database
- mapper
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention discloses a system and a method for automatically and intelligently integrating isomerous data. The present invention is characterized in that the present invention makes use of a mapping device and a basal inner core to carry out automatic searching treatment for network is a network omerous data base and to create a network data source model and a host machine positioning index table, and the mapping device and the basal inner core are used for mapping an automatic data structure, a data dictionary and a data semanteme. The present invention provides a system and a method for automatically and intelligently integrating isomerous data for terminal products, which is suitable and desirable for enterprises, governments, industries and electronic commerce agencies, has the advantages of high safety, high efficiency, low cost and high availability and can be used for any network environment.
Description
Technical field
The present invention relates to a kind of network data interconnection technique field relevant, relate more specifically to a kind of autonomous intelligence heterogeneous data integrated system with isomeric data integrated technology and data mining technology.
Background technology
As everyone knows, the internet, the interconnected information communication network covering the whole world that provides is provided for Intranet and wide area network three, but network only provides and has passed through with TCP/IP (transmission controlprotocol/internet protocol at present, transmission control and Internet protocol agreement) be master's computing equipment and being connected of soft port, in various hardware platforms, can realize between the different operating system that file-level is shared, as the file in the Intranet, driver shared device etc. and the FTP in the internet (file transfer protocol, file transfer protocol (FTP)) system.Even but be a technological gap to the direct interconnected and integrated of data for data between the various data structures.Because the influence of historical factor up to now, exists all types of Database Systems of preserving a large amount of structural datas and surpass kind more than 30 in network, and the file layout of preservation unstructured data is above ten thousand kinds.Owing to lack the interface of recognition data and unified data representation standard mutually, therefore,, exist extremely serious information island phenomenon between these isomeric datas though place oneself in the midst of among the global information interconnected network network.Produced great inhibition for the efficient information operating that is structured on the shared basis of extensive information.
Current, comprise Sybase manufacturer such as IBM, Oracle and be that the application software developer of representative has given the attention of height to the distributed data integrated technologies based on three nets with CA, and drop into suitable strength exploitation related software product.But such related software product that occurs is simple aid software at present, does not possess opening, independence and intelligent feature.Summing up its major defect has: not open; Not general; Not independent; There is not intelligence; Dangerous; Lack simultaneously tenability to Intranet, internet and wide area network; Lack the autonomous operation ability.
Summary of the invention
The purpose of this invention is to provide a kind of autonomous intelligence heterogeneous data integrated system, this system is a kind of in internet, LAN (Local Area Network) and wan environment, has opening, autonomy-oriented and the exploration of intelligentized isomeric data, data model structure analysis, data mining, semantic analysis, and the data integrated system of the cross-platform heterogeneous database of across a network of XML data transmission and processing ability.
The present invention is achieved in that
A kind of autonomous intelligence heterogeneous data integrated system, this system is a kind of in internet, LAN (Local Area Network) and wan environment, have opening, autonomy-oriented and the exploration of intelligentized isomeric data, data model structure analysis, data mining, semantic analysis, and the data integrated system of the cross-platform heterogeneous database of across a network of XML data transmission and processing ability, this system comprises:
A basic kernel of having realized that objectification database and a plurality of heterogeneous database drive, this kernel provides algorithm, rule, model, the bottom of said apparatus in the integrated processing of isomeric data to call and equipment control;
A mapper that is connected to above-mentioned basic kernel, this mapper realize that the network data in the integrated processing of isomeric data is explored and relationship map is handled;
A delver that is connected to above-mentioned basic kernel, this delver is realized data query engine, the data mining engine in the integrated processing of isomeric data, carries out network data excavation at the output result-mapping graph of above-mentioned mapper;
An interchanger that is connected to above-mentioned basic kernel, this interchanger are at the output-datagram of above-mentioned delver, and data formatting, the communication carried out in the integrated processing of isomeric data connect and management, and network control and data transmission;
A receiver that is connected to above-mentioned basic kernel, in the integrated processing procedure of isomeric data, this receiver at network carry out communication monitor, to above-mentioned interchanger reply, Data Receiving and database data push and upgrade.
Compared with prior art, the present invention possesses following advantage: open architecture; Dynamic application model; Support three net environment comprehensively; With the irrelevant application cross-platform ability of application; Support all types database automatically, intelligent data is integrated; Support all types destructuring character stream data structure automatically, intelligent data is integrated.
Description of drawings
According to following accompanying drawing and best enforcement description, characteristic of the present invention and advantage be easy to understand more.
Fig. 1 is the overall module map of autonomous intelligence heterogeneous data integrated system of the present invention;
Fig. 2-the 6th, the detailed block diagram of 5 modules among diagram Fig. 1;
Fig. 7 realizes general steps process flow diagram of the present invention;
Fig. 8-the 9th, the detailed step process flow diagram of automatic exploration and isomery mapping among diagram Fig. 7.
Embodiment
In Fig. 1, the present invention includes basic kernel 19, with the mapper 20 that is connected by basic kernel 19, delver 21, interchanger 22 and receiver 23.
Connect by data bus between mapper 20, delver 21, interchanger 22 and the receiver 23, mainly realize data internal transmission, message and the visit of shared data storehouse.
Fig. 2 illustrates the internal module structure of basic kernel 19 in more detail.Basis kernel 19 comprises common class interface 24, synchronization call device 25, asynchronous call device 26, file and database service engine 27, network communication engine 33 and Network protocol engine 35.Wherein, Network protocol engine comprises HTTP (hypertext transfer protocol, HTML (Hypertext Markup Language)) engine 38, ASP (active server page, dynamic state server homepage) engine 39, XML engine 40 and message queue 41 35 inner compositions; The network communication engine 3 comprises Network protocol engines such as communication driver 31 and TCP/IP engine 343 inner compositions.Translate service code in the middle of when the common class interface 24 in basic kernel 19 is embodied as one group of dynamic link libraries and operation, carry out run time call, control and management by synchronization call device 25 direct-connected and asynchronous call device 26 with it.File that is connected with common class interface 24 and database service engine 27 provide local system level file and objectification database service.The communications protocol engine 33 that is connected with common class interface 24 provides can be based on the Communications service on the multiple extendible network communication protocol, the drive software of the new hardware of continuous implantation, equipment is provided.Form by multiple kinds engines such as communication driver 31 and TCP/IP engines.The Network protocol engine 35 that is connected with common class interface 24 is to simplify the middleware services container in client, the service role each other in when operation.HTTP service, ASP service and XML service 40 are provided; And message queue service.Above-mentioned four services are provided by the HTTP engine 38 of forming Network protocol engine 35, ASP engine 39, XML engine 40 and message queue 41 respectively.
Fig. 3 illustrates the internal module structure of mapper 20 in more detail.Mapper 20 comprises SQL (structured query language, Structured Query Language (SQL)) model bank 43, SQL engine 42, sample mapper 53, relationship map device 52 and extracts mapper 51.Wherein, the sample mapper comprises database probe 63, model management device 62 and sample management device 61 53 inner compositions; The relationship map device comprises View Manager 60, algorithm management device 59 and connection manager 58 52 inner compositions; Extract mapper and comprise template engine 54, trigger engine 55, character converter 56 and data representation device 57 51 inner compositions.
SQL engine 42 in the mapper 20 is realized SQL definition, the storage and maintenance based on the relationship map language, and based on the analysis and the establishment of SQL standard.The SQL model bank 43 that is connected with SQL engine 42 provides SQL model and objects services, comprises containing SQL standard speech Faku County and SQL canonical function and computing storehouse.The sample mapper 53 that is connected with SQL engine 42 is realized automatic and the sample analysis of intelligent data source, exploration, structure establishment, maintenance and the acquisition scheme of diverse network are downloaded and safeguarded.Database probe 63 is wherein finished network data and is derived from moving the exploration; Model management device 62 is finished the network data source and is analyzed and manage; Sample management device 61 is finished the download and the maintenance of composition of sample establishment, maintenance and acquisition scheme.The relationship map device 52 that is connected with SQL engine 42 is hinted obliquely at the data integration relation that the output (connection among Fig. 3 between sample mapper 53 and the relationship map device 52) of sample mapper 53 is implemented between the data source composition of sample and the sampling plan structure.View Manager 60 managing virtual view object wherein; Algorithm management device 59 management SQL algorithm objects; Transmission annexation between connection manager 58 managing virtual view object and the SQL algorithm object.The extraction mapper 51 that is connected with SQL engine 42 is at the output of relationship map device 52, is implemented in the record every trade decimation rule definition on the definition list level data field row relationship map figure basis.Template engine 54 supplying a model predefine decimation rules wherein are beneficial to user's quick definition and the decimation rule definition that is implemented on relationship map device 52 bases; Trigger engine 55 realizes that to dynamic data base integrated and dynamic data extracts in the rules code of trigger section to source database system and implants; Character converter 56 is realized the automatic conversion at internationalization character set in the various Database Systems; Data representation device 57 provides the integrated result of visual database to browse in advance and is used to extract mapper and carries out the basic data structure service that autonomous intelligence is optimized.
Fig. 4 illustrates the internal module structure of delver 21 in more detail.Delver 21 comprises many main frames switching engine 64, data buffering pond 65, database-driven device 77, excavates trigger 76 and excavates actuator 75.Wherein, the inner driving interface that comprises at various Sybase system of forming of database-driven device 77; Excavate trigger and comprise event trigger 83, reciprocal counter 84 and start-up by hand device 85 76 inner compositions; Excavate actuator and comprise excavation filtrator 82, data reducer 81 and process tracking device 79 76 inner compositions.
Main frame switching engine 64 in the delver 21 realize mining process scheduling, priority arbitration, management of process, in the face of many main frames location, integrated flow analysis, the SQL of network environment translate, create and dispose and dynamically the communal space manage.The data driver 77 that is connected with main frame switching engine 64 provides the driving interface of predefined multiple Sybase to main frame switching engine 64; The excavation trigger 76 that is connected with main frame switching engine 64 realizes that excavation starts trigger model, provides start-up by hand, affairs driving startup and countdown to start three kinds of startups different or hybrid mode realization mining process and triggers.Event trigger 83 wherein, reciprocal counter 84 and start-up by hand device 85 are corresponding respectively and provide above-mentioned three types startup to trigger service.The excavation actuator 75 that is connected with main frame switching engine 64 is realized the data mining process according to sampling plan and SQL expression.Combined filtration model and algorithm that excavation filtrator 82 is wherein realized based on field contents; Data reducer 81 realizes that all throw in the output of operation process in the filtrator 82 to excavating, arrangement, processing on data item → data line → tables of data → 4 data ranks of result set difference; The process content exposure that process tracking device 79 is realized based on the process of excavation provides mining structure and excavates conclusion with patterned way.
Fig. 5 illustrates the internal module structure of interchanger 22 in more detail.Interchanger 22 comprises transmission and switching engine 86, formatter 95, encryption equipment 96, compressor reducer 97, delivery device 99, certificate server 94 and handgrip 95.
Fig. 6 illustrates the internal module structure of receiver 23 in more detail.Interchanger 22 comprises intercepts engine 110, formatter 115, decipher 116, separates the device 117 that contracts, audiomonitor 119, buffered data storehouse 114 and implanting device 118.
Fig. 7 has illustrated overall process sequence of steps of the present invention.
In step 101, carry out the automatic exploration of network heterogeneous database by basic kernel 19 and mapper 20 and handle; Create data source model, main frame location concordance list under the specified network environment, and the basis and the model parameter of the mapping of network isomeric data are provided for follow-up step 102.
In step 102, by mapper 20 and delver 21 on the basis of step 101 output model and parameter, carry out the mapping of data structure, data dictionary and the data semantic of robotization, generate and be provided at the execution command of carrying out data mining and extraction in subsequent step 103 data minings.
In step 103, excavate and extract under the prerequisite that executes instruction in step 102 output by delver 21, carry out data mining at a plurality of heterogeneous database spanning networks, generate the data to be transferred newspaper.
In step 104, accept the output of step 103 by interchanger 22, pack, encrypt, after a series of processing such as compression, by network monitoring mechanism the datagram on the network is detected and accepted and finally finish the data of target database are inserted and renewal by receiver 23.
Fig. 8 has more in depth illustrated the detailed diagram sequence of steps of step 101 among Fig. 7.
By step 301 stem, obtain MaskIP (mask internet protocol address, internet address mask), verify that Intranet communication state, network area are discriminated automatically and survey and initialization network communication link; Afterwards, in the memory field, create main frame by step 302 and explore formation and indicator index, finish the initialization of target database being explored storehouse; Drop into step 303 by single process, lock each main frame and explore address in formation, at first finish OS Type that it is moved and relevantly move, the automatic exploration of configuration parameter; According to a certain OS Type (for example Windows and UNIX etc., subsequent processing steps is with different), in step 304 by single process loading operation system service model (from the built-in model bank of intelligent information pump, the function service realizes with the form of open plug-in unit), and according to the operating system service model scan service port that loads; Step 305 is that each port that captures is carried out automatic comparison with the database service model; Judge whether on certain main frame port, to capture the service that is complementary with certain database service model by step 306; If then will catch parameter and write target database exploration storehouse by step 307; If not, then on earth by step 308 test pointer; In 309 steps, judge whether on earth; If not, then show the database service model that still existence is not compared as yet, at this moment, will transfer to the detection that step 305 continues; If, then finish the transfer of storehouse to formation by step 310, each has verified the processing such as initialization that the inner specifically step of data object of database service is finished formation for follow-up scanning; 5 multitasking forms that step is a parallel thread of from 312 to 316, at first finish establishment by step 311 to the Management Controller of parallel thread, and set up the sweep parameter shared drive simultaneously, this shared drive will be realized the mutual and communication of the dynamic data between the parallel thread; Certain thread of step 312 startup carries out the detection at the internal data example of certain database service in the formation; The result by the shared drive communication submit to an other thread by step 313 finish with database search model storehouse in the checking of object; Step 314 is judged, if do not match, then write the suitable data of description structure of shared drive by step 315, if the coupling would directly enter step 316 test queue pointer, if do not reach the formation tail end as yet, then control is returned step 313, locate and detect other data model objects; If reached the formation tail end, then enter step 317, finish parallel thread, reclaim resource, controller waits returning of whole threads; In case all return, in step 318, finish according to the contribution internal storage data structure that has been created in the shared drive by controller, write or the database search model storehouse of updating system.
Fig. 9 has more in depth illustrated the detailed diagram sequence of steps of step 102 among Fig. 7.
Fig. 9 has illustrated according to of the present invention, the diagram sequence of steps that the network isomeric data relational model that is carried out with mapper 20 shines upon.In step 351, the response that mapper is sent according to receiver 23, automatic dynamic is loaded the excavation mapping object module that receiver 23 pushes shared drive, and historical conclusion sample is compared; Whether in step 352, judge and the history coupling, the part of coupling is called kernel by common interface carry out loading the refresh process of well, and directly in step 354, finish; To not matching partly or entirely of existing, precursor index tree structure is set up in initialization relationship map workspace in step 355; Subsequently, in step 356, transfer to common interface on the one hand and excavate processing of loading well and end in step 357 again; On the other hand, in step 358,, create or load existing discrete relationship mapping ruler tree according to excavating mapping demand and precursor index tree; In step 359, mapper 20 carries out excavating the parsing of mapping target, and separates the discrete mapping theme that excavates, and sets up discrete theme in shared drive; In step 360, mapper 20 is created and the corresponding multi-thread concurrent Management Controller of discrete theme; In step 361, create and the corresponding processing threads of each discrete theme by the multi-thread concurrent Management Controller, carry out theme thread discrete programming and excavate mapping; In step 362, differentiate whether successfully form theme mapping conclusion, if thread is then reclaimed in success in step 364, merge rule, and finish the dynamic integrity of each theme being excavated mapping; In step 365, mapper 20 is allotted algorithm, function model object according to integrated mapping conclusion, generates recordable instantiation and excavates mapping model; In step 366, mapper is finished excavating the marking of mapping model, implants map tag in database source; In step 367, according to the excavation mapping model after the marking, call kernel services in the common interface by mapper, carry out pre-data mining and extract, in hard disc apparatus, create the local data base structured data and extract set; In step 368,, whether satisfy and on what degree, satisfy the excavation demand to the data acquisition realization property analysis of pre-extraction; In step 369, carry out performance and efficiency analysis, and finish active optimization at realization property and performance; Through after n iteration in step 370, the equipped well line data stores processor of going forward side by side of carrying is initiatively reported the demand equipment of data excavation simultaneously and is replied.
Be to be understood that the front is a diagram principle of the present invention, those skilled in the art do not depart from the scope of the present invention and essence situation under can carry out various modifications.
Claims (3)
1. autonomous intelligence heterogeneous data integrated system, this system is a kind of in internet, LAN (Local Area Network) and wan environment, have opening, autonomy-oriented and the exploration of intelligentized isomeric data, data model structure analysis, data mining, semantic analysis, and the data integrated system of the cross-platform heterogeneous database of across a network of XML data transmission and processing ability, it is characterized in that this system comprises:
A basic kernel of having realized that objectification database and a plurality of heterogeneous database drive, this kernel provides algorithm, rule, model, the bottom of said system in the integrated processing of isomeric data to call and equipment control;
A mapper that is connected to above-mentioned basic kernel, this mapper realize that the network data in the integrated processing of isomeric data is explored and relationship map is handled;
A delver that is connected to above-mentioned basic kernel, this delver is realized data query engine, the data mining engine in the integrated processing of isomeric data, carries out network data excavation at the output result-mapping graph of above-mentioned mapper;
An interchanger that is connected to above-mentioned basic kernel, this interchanger are at the output-datagram of above-mentioned delver, and data formatting, the communication carried out in the integrated processing of isomeric data connect and management, and network control and data transmission;
A receiver that is connected to above-mentioned basic kernel, in the integrated processing procedure of isomeric data, this receiver at network carry out communication monitor, to above-mentioned interchanger reply, Data Receiving and database data push and upgrade.
2. autonomous intelligence heterogeneous data integrated system according to claim 1, it is characterized in that, connect by data bus between described mapper, delver, interchanger and the receiver, to realize the data internal transmission, message and the visit of shared data storehouse.
3. autonomous intelligence heterogeneous data integrated system according to claim 1, it is characterized in that, described basic kernel comprises common class interface, synchronization call device, asynchronous call device, file and database service engine, network communication engine and Network protocol engine, translate service code in the middle of when the common class interface in basic kernel is embodied as one group of dynamic link libraries and operation, carry out run time call, control and management by synchronization call device direct-connected and asynchronous call device with it.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB021489912A CN1152334C (en) | 2002-11-18 | 2002-11-18 | Autonomous intelligent isomeri data integration system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB021489912A CN1152334C (en) | 2002-11-18 | 2002-11-18 | Autonomous intelligent isomeri data integration system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1410915A CN1410915A (en) | 2003-04-16 |
| CN1152334C true CN1152334C (en) | 2004-06-02 |
Family
ID=4751598
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB021489912A Expired - Fee Related CN1152334C (en) | 2002-11-18 | 2002-11-18 | Autonomous intelligent isomeri data integration system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN1152334C (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100401288C (en) * | 2005-05-30 | 2008-07-09 | 北京慧讯信息技术有限公司 | Distributed data source data integration system and method |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2006026686A1 (en) * | 2004-08-31 | 2006-03-09 | Ascential Software Corporation | User interfaces for data integration systems |
| CN100401287C (en) * | 2004-09-16 | 2008-07-09 | 北京慧讯信息技术有限公司 | Chinese data intelligent classifying system and method |
| US9473265B2 (en) * | 2005-12-22 | 2016-10-18 | Qualcomm Incorporated | Methods and apparatus for communicating information utilizing a plurality of dictionaries |
| CN1845104B (en) * | 2006-05-22 | 2012-04-25 | 赵开灏 | System and method for intelligent retrieval and processing of information |
| CN104820700B (en) * | 2015-05-08 | 2018-07-20 | 中国南方电网有限责任公司电网技术研究中心 | The processing method of substation's unstructured data |
| EP3260991A1 (en) * | 2016-06-20 | 2017-12-27 | Siemens Aktiengesellschaft | System and method configured to execute data model transformations on data for cloud based applications |
| CN114070768B (en) * | 2021-11-29 | 2023-11-03 | 中国工商银行股份有限公司 | Penetration test method, penetration test device, computer equipment and storage medium |
-
2002
- 2002-11-18 CN CNB021489912A patent/CN1152334C/en not_active Expired - Fee Related
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100401288C (en) * | 2005-05-30 | 2008-07-09 | 北京慧讯信息技术有限公司 | Distributed data source data integration system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN1410915A (en) | 2003-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101553769B (en) | Method and system for tracking and monitoring computer applications | |
| US6434558B1 (en) | Data lineage data type | |
| Brown | Overview of SciDB: large scale array storage, processing and analysis | |
| EP0625756B1 (en) | Federated information management architecture and system | |
| EP2182448A1 (en) | Federated configuration data management | |
| CN1213379C (en) | Real-time detabase object statistics collection | |
| CN108762900A (en) | High frequency method for scheduling task, system, computer equipment and storage medium | |
| CN1152334C (en) | Autonomous intelligent isomeri data integration system and method | |
| US7680813B2 (en) | Information management system | |
| US20060129609A1 (en) | Database synchronization using change log | |
| US7496901B2 (en) | Method for boundary trace with reproduction facility | |
| CN109213752A (en) | A kind of data cleansing conversion method based on CIM | |
| CN108647886B (en) | Scientific computing process management system | |
| CN113347170A (en) | Intelligent analysis platform design method based on big data framework | |
| Ataei et al. | NeoMycelia: A software reference architecturefor big data systems | |
| CN1152338C (en) | Parallel distributed-data base processing method and device | |
| US20100145921A1 (en) | Information management inventory | |
| Gadepally et al. | Version 0.1 of the bigdawg polystore system | |
| CN1257463C (en) | System for automatic monitoring image management and communication | |
| CN110245037B (en) | Hive user operation behavior restoration method based on logs | |
| Gadepally et al. | BigDAWG version 0.1 | |
| CN1737795A (en) | Method for data digging and knowledge discovery under multi data source cooperation condition | |
| US20170322973A1 (en) | System and Method to Optimize Queries on a View | |
| CN1393771A (en) | Method for quickly eliminating different information in database | |
| Schnase et al. | HB1: Initial design and implementation of a hyperbase management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20040602 Termination date: 20111118 |