CN107844378A - A kind of Distributed Heterogeneous Data resources integration and management system - Google Patents
A kind of Distributed Heterogeneous Data resources integration and management system Download PDFInfo
- Publication number
- CN107844378A CN107844378A CN201610836113.4A CN201610836113A CN107844378A CN 107844378 A CN107844378 A CN 107844378A CN 201610836113 A CN201610836113 A CN 201610836113A CN 107844378 A CN107844378 A CN 107844378A
- Authority
- CN
- China
- Prior art keywords
- data
- message
- information
- module
- configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/544—Remote
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Abstract
The present invention relates to a kind of Distributed Heterogeneous Data resources integration and management system, including data acquisition configuration tool and Heterogeneous Information collection adaptation services two parts；Data acquisition adaptive tool provides the user the operation interface of close friend, realizes effective configuration to data endpoint configurations, data format and attribute configuration, and transformation rule；The data message that Heterogeneous Information collection adaptation services primary recipient data publisher sends over, a pipeline is distributed for each external system, handles different types of information by different message channels, message is handled using push mode.
Description
Technical field
The invention belongs to technical field of the computer network, and in particular to a kind of Distributed Heterogeneous Data resources integration and management are
System.
Background technology
Various data exchanges and data access problem are faced during the Internet of Things Information System configuration of smart city.It is and every
The data of type are required for stand-alone development to integrate service software to realize the access of the partial data because of its form, agreement difference
And transmission.
Carrying out method that is integrated and managing to Distributed Heterogeneous Data resource at present mainly has data copy method and set of patterns
Into method.Data copy method refers in the data duplication of each data source to relative other data sources, and safeguards
Data source uniformity on the whole, improve the efficiency that information sharing utilizes.Most common data copy method is exactly to pass through ETL framves
Structure, by extracting, changing, loading procedure, by the data converted the method that increment or whole import data warehouse according to plan.This
The advantages of kind data integration mode is data high concentration, and it is more convenient to establish client.Shortcoming to propose one it is unified
Database schema meets the needs of multiple applications.And when phase is continually read and changed in multiple applications by shared data bank
Same data, can bring performance bottleneck.The basic thought of Mode integrating method is, when building integrated system by each data source
Data View is integrated into global schema, allows users to pellucidly access the data of each data source according to global schema.Set of patterns
Include federative database and middleware method into method.
Typically the data integrated system based on middleware mainly includes middleware and wrapper, wherein each data source pair
A wrapper is answered, middleware is interacted by wrapper and each data source.User sends out on the basis of global schema to middleware
Go out inquiry request.Middleware pattern can integrate the data source of non-database form, real-time.But perceived for Internet of Things
For situation more than data source category, it is necessary to develop different wrappers respectively, autgmentability is not strong.
The content of the invention
The purpose of the present invention is to propose to a kind of Distributed Heterogeneous Data resources integration and management system, it is directed to by flexible
Configuration tool, code development can not needed or in the case of making a small amount of modification of program, realize that real-time perception data are believed
Collection, conversion and the distribution of breath.
Technical scheme is as follows：
A kind of Distributed Heterogeneous Data resources integration and management system, it is characterised in that：Including data acquisition configuration tool and
Heterogeneous Information gathers adaptation services two parts；Data acquisition adaptive tool provides the user the operation interface of close friend, realizes logarithm
According to effective configuration of endpoint configurations, data format and attribute configuration, and transformation rule；Heterogeneous Information collection adaptation services are main
The data message that data publisher sends over is received, a pipeline is distributed for each external system, passes through different message
The different types of information of passage processing, is handled message using push mode.
The data end points, i.e. data publisher and data subscriber；The transformation rule configuration, include the number of data source
Configured according to form, the configuration of target data form, and the data conversion rule configuration between them.
Heterogeneous Information collection adaptation services include information router module, data distribution module, link management module, different
Normal message processing module, data mapping relations management module and running state monitoring module, wherein, information router module is according to disappearing
Cease the route of type completion message；Data distribution module forwards the message to corresponding subscriber；Link management module management chain
The connection on road, closing, abnormal information, established by distributing unique ID marks to each end points between publisher and subscriber
Annexation；Abnormal information processing module handles abnormal information；Data adaptation interface module carries out message polymerization and decomposition, simultaneously
Data processing module is called to carry out Message Processing；Data mapping relations management module management information issuing person and consumer passage；
Running state monitoring module checks each channel status information, and counts the quantity of publisher and subscriber.
Of the invention mainly to realize message format adaptation, message data format conversion, message transfer function, it primarily serves different
The access of structure real-time perception data, standardize and share.Specifically solve three problems：1st, when having multiple data sources and goal systems
When, how message channel is established between the system of isomery, and route messages to goal systems.2nd, disappearing data source systems
Breath form is converted into the data mode of reference format.The 3rd, friendly operation interface is provided, user is configured publisher's system
Communication protocol, network communicated information with subscriber's system, and the form of source data and target data.
Embodiment
The Distributed Heterogeneous Data resources integration and management system of the present invention includes data acquisition configuration tool and Heterogeneous Information
Adaptation services two parts are gathered, two parts functional module mainly realizes information exchange by configuration file, and then realizes isomery sense
Know the collection and standardization of information.
Isomeric data accesses and knot of the whole design concept of standardized system based on SOA Technical Architectures Yu software pipeline infrastructure
Close, be separated using enriched data acquisition channel and user service role (data publisher, data subscription person), realize collection
The flexible configuration of passage and user role, back-end data handling process is designed using software pipeline infrastructure, according in information
Content, data processing module is transmitted data to by pipeline and is parsed and is handled, translated data is distributed to difference
Subscriber.
Data acquisition adaptive tool provides the user the operation interface of close friend, realizes to the configuration of data collection terminal point, number
According to effective configuration of form and attribute configuration, and transformation rule.Data end points is data publisher and data subscriber.Conversion
Rule configuration includes the data format configuration of data source, the configuration of target data form, and the data conversion rule between them
Configuration.
The data message that Heterogeneous Information collection adaptation services primary recipient data publisher sends over, is outside each
System distributes a pipeline, and different types of information is handled by different message channels, and message is carried out using push mode
Processing.
Data-pushing refers to that data source is actively pushed data into purpose system.By data processing, it is converted into marking
The information of quasiconfiguaration is distributed to data subscription person's (including database middleware).
It is asynchronous between coordination data publisher and subscriber, establish and cache at subscriber end.
Heterogeneous Information collection adaptation services include information router module, data distribution module, link management module, abnormal letter
Cease processing module, data mapping relations management module and running state monitoring module.Information router module is complete according to type of message
Into the route of message.Data distribution submodule forwards the message to corresponding subscriber.The company of link management module management link
Connect, close, abnormal information, connection pass is established between publisher and subscriber by distributing unique ID marks to each end points
System.Abnormal information processing module handles abnormal information.Data adaptation interface module carries out message polymerization and decomposition, while calls number
Message Processing is carried out according to processing module.Data mapping relations management module management information issuing person and consumer passage.Run shape
State monitoring module checks each channel status information, and counts the quantity of publisher and subscriber.
User of service realizes the work(such as systematic parameter, data protocol form, data protocol conversion by interface visualization mode
Can operation, system according to the automatic acquisition of scientific information of configuration to network environment, message semantics format information, from Various types of data publisher
(data source) receives data, and by the process such as filtering, parsing, corresponding data subscription person is routed to after being handled by data platform
(data whereabouts), and then realize the shared of data.Data platform of the system as data conversion route service, in different issues
Mapping relations are established between person and subscriber, corresponding processing is carried out for the data processing type of configuration, avoids multiple packagings
The exploitation of device, realizes code reuse.
Claims (3)
1. a kind of Distributed Heterogeneous Data resources integration and management system, it is characterised in that：Including data acquisition configuration tool and different
Structure information gathering adaptation services two parts；Data acquisition adaptive tool provides the user the operation interface of close friend, realizes to data
Endpoint configurations, data format and attribute configuration, and effective configuration of transformation rule；Heterogeneous Information collection adaptation services mainly connect
The data message that data publisher sends over is received, a pipeline is distributed for each external system, is led to by different message
The different types of information of road processing, is handled message using push mode.
2. Distributed Heterogeneous Data resources integration according to claim 1 and management system, it is characterised in that：The data terminal
Point, i.e. data publisher and data subscriber；The transformation rule configuration, include the data format configuration of data source, number of targets
Configured according to form, and the data conversion rule configuration between them.
3. Distributed Heterogeneous Data resources integration according to claim 1 and management system, it is characterised in that：The isomery letter
Breath collection adaptation services include information router module, data distribution module, link management module, abnormal information processing module, number
According to mapping relations management module and running state monitoring module, wherein, information router module is according to type of message completion message
Route；Data distribution module forwards the message to corresponding subscriber；It is the connection of link management module management link, closing, different
Normal information, established a connection by distributing unique ID marks to each end points between publisher and subscriber；Abnormal letter
Cease processing module processing abnormal information；Data adaptation interface module carries out message polymerization and decomposition, while calls data processing mould
Block carries out Message Processing；Data mapping relations management module management information issuing person and consumer passage；Running state monitoring mould
Block checks each channel status information, and counts the quantity of publisher and subscriber.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610836113.4A CN107844378A (en) | 2016-09-21 | 2016-09-21 | A kind of Distributed Heterogeneous Data resources integration and management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610836113.4A CN107844378A (en) | 2016-09-21 | 2016-09-21 | A kind of Distributed Heterogeneous Data resources integration and management system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107844378A true CN107844378A (en) | 2018-03-27 |
Family
ID=61656990
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610836113.4A Pending CN107844378A (en) | 2016-09-21 | 2016-09-21 | A kind of Distributed Heterogeneous Data resources integration and management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107844378A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109150610A (en) * | 2018-08-29 | 2019-01-04 | 深圳市未来交互信息技术有限公司 | The network event acquisition method of rule-based adaptation |
| CN109901992A (en) * | 2019-01-18 | 2019-06-18 | 竞技世界（北京）网络技术有限公司 | A kind of method of Remote Dynamic oracle listener process performing |
| CN111198915A (en) * | 2019-12-26 | 2020-05-26 | 浙江中控技术股份有限公司 | Data distribution system and method based on public information model |
| CN113377710A (en) * | 2020-03-10 | 2021-09-10 | 中国移动通信集团湖南有限公司 | Data acquisition method and device and electronic equipment |
-
2016
- 2016-09-21 CN CN201610836113.4A patent/CN107844378A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109150610A (en) * | 2018-08-29 | 2019-01-04 | 深圳市未来交互信息技术有限公司 | The network event acquisition method of rule-based adaptation |
| CN109150610B (en) * | 2018-08-29 | 2021-05-04 | 深圳市未来交互信息技术有限公司 | Network event acquisition method based on rule adaptation |
| CN109901992A (en) * | 2019-01-18 | 2019-06-18 | 竞技世界（北京）网络技术有限公司 | A kind of method of Remote Dynamic oracle listener process performing |
| CN109901992B (en) * | 2019-01-18 | 2022-03-04 | 竞技世界（北京）网络技术有限公司 | Method for remotely and dynamically monitoring program execution behavior |
| CN111198915A (en) * | 2019-12-26 | 2020-05-26 | 浙江中控技术股份有限公司 | Data distribution system and method based on public information model |
| CN113377710A (en) * | 2020-03-10 | 2021-09-10 | 中国移动通信集团湖南有限公司 | Data acquisition method and device and electronic equipment |
| CN113377710B (en) * | 2020-03-10 | 2023-08-01 | 中国移动通信集团湖南有限公司 | Data acquisition method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105897797A (en) | Massive heterogeneous data access and standardization system | |
| CN107844378A (en) | A kind of Distributed Heterogeneous Data resources integration and management system | |
| CN102202093B (en) | A kind of method and system realizing transducer adaptation | |
| CN102710435B (en) | System integration interface adapter based on common information model (CIM) | |
| CN102307223B (en) | A kind of method and system realizing application platform adaptation | |
| CN101697166B (en) | Method for accelerating data integration of heterogeneous platform | |
| CN105635283A (en) | Organization and management and using method and system for cloud manufacturing service | |
| US20110085461A1 (en) | Flexible network measurement | |
| CN101794409A (en) | Real-time data exchange model of cross-platform application system | |
| CN103870274A (en) | Visual development assembly for realizing data interaction between heterogeneous systems based on web service technology | |
| CN101739460A (en) | Grid-based spatial data source unification service system and method | |
| US20160277525A1 (en) | Method and controller for clustering applications in a software-defined network | |
| CN107276992A (en) | A kind of method and system of multi-data source access | |
| CN108446335A (en) | Heterogeneous system data pick-up based on database and unified outbound data exchange method | |
| CN106612335B (en) | The method of the information exchange and communication of IoT is realized using Docker container | |
| CN104702638A (en) | Event subscribing and dispatching method and device | |
| CN107622064A (en) | A kind of method for reading data and system | |
| CN101576922B (en) | Access system and access method | |
| CN113949725A (en) | Cloud edge cooperation-based manufacturing resource access platform and method | |
| CN101594263B (en) | System for monitoring network communication data packets | |
| CN101110858A (en) | Telecommunication report generation system and method thereof | |
| CN103092982B (en) | Based on the information service system of multi views business model | |
| CN102629345B (en) | Chain type links up collaboration method, Apparatus and system | |
| CN106789965A (en) | A kind of Internet of Things data exchange method and system | |
| CN104219280B (en) | A kind of intelligent use data transmission channel |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180327 |
|
| WD01 | Invention patent application deemed withdrawn after publication |