CN111400382A - Model-driven data integration middleware and implementation method - Google Patents
Model-driven data integration middleware and implementation method Download PDFInfo
- Publication number
- CN111400382A CN111400382A CN202010145552.7A CN202010145552A CN111400382A CN 111400382 A CN111400382 A CN 111400382A CN 202010145552 A CN202010145552 A CN 202010145552A CN 111400382 A CN111400382 A CN 111400382A
- Authority
- CN
- China
- Prior art keywords
- data
- model
- management
- platform
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a model-driven data integration middleware and an implementation method thereof, wherein the model-driven data integration middleware comprises an information model management module, a storage model management module, a heterogeneous data integration module and a general exchange service management module; the information model management module is used for value domain code management, data element management and data set management; the storage model management module is used for data source management, theme library management and storage model management; the heterogeneous data integration module is used for realizing two integration modes, timing integration, real-time integration and the like; the invention realizes the integrated management of standard specification management, model establishment, data center establishment, interface specification management, data storage, sharing and exchange service interface in the data integration process, defines the models of an information model, a storage model, an exchange model and the like, and drives the automation and the tool realization of the whole data integration process through the models, thereby reducing the upgrading and updating cost of the system and improving the flexible expansibility of the system.
Description
Technical Field
The invention relates to the technical field of data integration, in particular to a model-driven data integration middleware and an implementation method.
Background
The method has the advantages that data sharing and interconnection of all departments and all systems are enhanced, the goal of enabling data to run more and enabling people to run less is the important goal of information construction of all industries at present, data integration is the important link for achieving the goal, and the data integration relates to the content and a plurality of technical processing links such as data integration and exchange standard establishment, heterogeneous data standardization processing, data center model establishment, data storage rule management, data exchange standard and service generation. In the conventional data integration method, a certain part is usually realized by a plurality of discrete tools respectively, integrated management cannot be realized, and a plurality of defects exist in the aspects of systematization, automation and standardized management, and mainly comprise the following aspects:
(1) the data integration standard does not implement structured management. The standards for data integration may change as the industry develops and business needs change. The traditional data integration method generally manages the data integration standard through unstructured documents, and the upgrading of each version requires manual maintenance of the unstructured documents and then upgrading to a data center by means of other tools. The process is complicated, the support to a plurality of versions cannot be realized, and the updating and upgrading cost of the system is high.
(2) The data integration interface specification cannot be automatically generated by the system. Data integration needs to provide an integrated interface specification document for an integrated system, the traditional method needs manual maintenance, the situation that the interface specification is inconsistent with the specific interface implementation often occurs easily, the maintenance cost is high, and the effect is not ideal enough.
(3) The data storage model is not managed in the data integration system. Conventional data integration methods generally do not define the concept of a storage model, and database structures are generally completed using third party database modeling tools based on data exchange standards. Because the data integration middleware does not store database metadata information, the data integration and exchange are required to be finished by hard coding and cannot be realized by a configuration mode.
(4) The data storage rules do not implement program configuration management. The creation of the data center table from the data integration standard needs to be completed by a third-party modeling tool, the data integration middleware cannot store the metadata information of the database, and the data storage rule and the encryption rule are generally completed by hard coding or are additionally realized by developing a configuration management system.
(5) The data exchange interface cannot be dynamically generated. The traditional data integration middleware is generally realized by adopting a hard coding mode according to requirements, and has high updating cost and insufficient flexible expansibility.
(6) The data exchange standard cannot realize the adaptation of various formats in a configuration mode. The data integration process often involves adaptation to various data exchange standards, the traditional method is generally realized through a hard coding mode, the upgrading and updating cost is high, and the flexibility and expansibility are not enough.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a model-driven data integration middleware and a realization method, which realize the integrated management of standard specification management, model establishment, data center establishment, interface specification management, data storage, sharing and exchange service interface in the data integration process, define models such as an information model, a storage model and an exchange model, drive the automation and the tool realization of the whole data integration process through the model, reduce the updating cost of system upgrading and improve the flexible expansibility of the system.
The purpose of the invention is realized by the following technical scheme:
a model-driven data integration middleware comprises an information model management module, a storage model management module, a heterogeneous data integration module and a general exchange service management module; the information model management module is used for value domain code management, data element management and data set management; the storage model management module is used for data source management, theme library management and storage model management; the heterogeneous data integration module is used for realizing two integration modes, namely timing integration and real-time integration; and the universal exchange service management module is used for abstracting a storage object in the middle of the platform on the basis of the information model and the storage model, storing the transmitted data in a key value pair mode, and generating universal adding, modifying, deleting and inquiring services according to the data set which needs to be stored and exchanged and the structure of the storage model.
Furthermore, the value domain code management provides standardized code and non-standardized code standardized conversion processing for the application systems inside and outside the platform, and the mapping relation between related synonyms before the information exchange and docking of different systems, so that uniform value domain codes are provided for the whole platform, and a uniform standardized basis is provided for data exchange and sharing;
the data element management comprises data element classification management and data element detail management;
the data set management can generate an information model, an exchange specification, a data storage model and a storage specification of data exchange, and can manage data sets of different business domains by data set classification.
Furthermore, the data source management can perform the operations of adding, deleting, modifying and checking the data source, maintain the pool attribute of the data source and provide database connection words for a data acquisition tool;
the theme bank management is used for establishing classification for all storage models in the same service domain according to different service domains, so that management of a system physical model is facilitated, and meanwhile, the system operates to assign different modes or database examples for each theme bank and store the theme bank data in a database-by-database manner;
and the storage model management is used for managing a storage model table, the storage model table is used for selecting a corresponding data set from an information model resource library as required to be associated according to the information model in a specific service scene, and appointing a physical table name, the storage model table supports infinite master-slave relation, and a corresponding physical table is generated according to the appointed physical table name.
Further, the timed acquisition is used for acquiring data by using CDC and ET L according to service data and index summarized data with low interactivity requirements;
the real-time acquisition is used for exchanging data by adopting a heterogeneous, multi-interface and stable message mechanism aiming at service data with high requirement on interaction urgency.
Further, the data element management includes standard code management and verification management.
A model-driven data integration middleware implementation method comprises the following steps:
s1, building a model driving data center;
s2, integrating data;
and S3, sharing data exchange.
Further, in step S1, the method includes:
s11, defining a corresponding information model including value domain codes, data elements and data set information through an integrated middleware configuration platform according to specific service requirements;
s12, creating a subject library according to business needs, generating a data storage model according to information model configuration, associating the data storage model with the subject library, generating a database table structure based on the storage model, completing creation of the database table, and establishing a complete data storage system, wherein the data storage system comprises constraint conditions of each data field, cleaning and conversion rules, whether encryption transmission and storage are needed or not can be set through a configuration platform.
Further, in step S2, the method includes:
s21, the access side creates a related standard interface library at the exchange node according to the standard interface library script of the data integration platform;
s22, the access side loads the service data of the information system related to the exchange node to the standard interface library of the data integration platform according to the standard specification of the data integration platform; the whole process is divided into two stages, wherein the first stage is to import all historical data into a standard interface library of the data integration platform in batches, and the second stage is to import newly generated incremental data into the standard interface library of the data integration platform every day;
s23, performing data quality evaluation on the data in the standard interface library of the data integration platform through a data quality evaluation tool to ensure that the data quality meets the relevant requirements of the data integration platform;
s24, the access party calls the data integration platform data interface service to realize data uploading through the data integration platform data interface service client;
s25, the data integration platform evaluates the data quality of the data uploaded by the data access party, and uploads the data to the data center of the data integration platform under the condition that the data quality meets the project requirements after evaluation; and under the condition that the data quality does not meet the project requirements, feeding the data quality report back to a data uploading unit, requiring the data uploading unit to be modified, and uploading the data again after meeting the requirements of the data integration platform.
Further, in step S3, the method includes:
s31, the data integration middleware generates data query interface service according to the data exchange and sharing interface specification configuration based on the information model and the storage model;
s32, the data exchange node is used for exchanging and sharing the interface service according to the data provided by the data integration platform and developing the client called by the related interface service according to the specification;
s33, calling information platform data exchange sharing service through the client, and finishing safety access control to the service and access authority authentication requested by the client by the information platform data exchange sharing service;
s34, the information platform data exchange sharing service realizes the query and acquisition of client request data by accessing the information platform data center, and assembles and returns the data in the format of interface specification;
and S35, the data exchange node acquires the information platform shared data returned by the information platform exchange for the service application of the data exchange node.
The invention has the beneficial effects that:
the invention realizes the integrated management of standard specification management, model establishment, data center establishment, interface specification management, data storage, sharing and exchange service interface in the data integration process, defines the models of information model, storage model, exchange model and the like, drives the automatic and tool realization of the whole data integration process through the models, reduces the updating cost of system upgrading, improves the flexible expansibility of the system, and particularly mainly realizes the following functions:
(1) the invention realizes the structured management of the data integration standard, carries out the structured management of the data integration standard from several aspects such as value domain codes, data elements, data sets and the like based on industry specifications and business requirements, forms an information model of data integration, and ensures that the data set standard can realize multi-version management.
(2) The invention realizes the automatic generation and export of the data interface specification, ensures the standard unified management of each access party and the central terminal, can realize the automatic generation and update of the data interface specification based on the information model, and greatly reduces the manual intervention process.
(3) The invention manages the metadata information of the database through the storage model, provides important support for data processing, exchange and sharing, generates the subject database and the data storage model based on the information model, and ensures that the adaptation of the data exchange standard and the generation of the universal exchange service can be realized.
(4) The invention can generate general data query service, storage service and the like based on the information model and the storage model, thereby greatly reducing the hard coding proportion, reducing the construction and implementation cost of the system and improving the flexible expansibility of the system.
(5) The invention can realize the adaptive conversion of various data exchange formats and the like based on the information model.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is an overall frame design of the present invention;
FIG. 2 is a technical architectural design of the present invention;
FIG. 3 is a data collection and aggregation process of the present invention;
fig. 4 is a data exchange sharing process according to the present invention.
Detailed Description
The technical solutions of the present invention are further described in detail below with reference to the accompanying drawings, but the scope of the present invention is not limited to the following. All of the features disclosed in this specification, or all of the steps of a method or process so disclosed, may be combined in any combination, except combinations where mutually exclusive features and/or steps are used.
Any feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving equivalent or similar purposes, unless expressly stated otherwise. That is, unless expressly stated otherwise, each feature is only an example of a generic series of equivalent or similar features.
Specific embodiments of the present invention will be described in detail below, and it should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that: it is not necessary to employ these specific details to practice the present invention. In other instances, well-known circuits, software, or methods have not been described in detail so as not to obscure the present invention.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Before describing the embodiments, some necessary terms need to be explained. For example:
if the terms "first," "second," etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a "first" element discussed below could also be termed a "second" element without departing from the teachings of the present invention. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements present.
The various terms appearing in this application are used for the purpose of describing particular embodiments only and are not intended as limitations of the invention, with the singular being intended to include the plural unless the context clearly dictates otherwise.
When the terms "comprises" and/or "comprising" are used in this specification, these terms are intended to specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence and/or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As shown in fig. 1 to 4, a model-driven data integration middleware includes an information model management module, a storage model management module, a heterogeneous data integration module, and a universal exchange service management module; the information model management module is used for value domain code management, data element management and data set management; the storage model management module is used for data source management, theme library management and storage model management; the heterogeneous data integration module is used for realizing two integration modes, namely timing integration and real-time integration; and the universal exchange service management module is used for abstracting a storage object in the middle of the platform on the basis of the information model and the storage model, storing the transmitted data in a key value pair mode, and generating universal adding, modifying, deleting and inquiring services according to the data set which needs to be stored and exchanged and the structure of the storage model.
Furthermore, the value domain code management provides standardized code and non-standardized code standardized conversion processing for the application systems inside and outside the platform, and the mapping relation between related synonyms before the information exchange and docking of different systems, so that uniform value domain codes are provided for the whole platform, and a uniform standardized basis is provided for data exchange and sharing;
the data element management comprises data element classification management and data element detail management;
the data set management can generate an information model, an exchange specification, a data storage model and a storage specification of data exchange, and can manage data sets of different business domains by data set classification.
Furthermore, the data source management can perform the operations of adding, deleting, modifying and checking the data source, maintain the pool attribute of the data source and provide database connection words for a data acquisition tool;
the theme bank management is used for establishing classification for all storage models in the same service domain according to different service domains, so that management of a system physical model is facilitated, and meanwhile, the system operates to assign different modes or database examples for each theme bank and store the theme bank data in a database-by-database manner;
and the storage model management is used for managing a storage model table, the storage model table is used for selecting a corresponding data set from an information model resource library as required to be associated according to the information model in a specific service scene, and appointing a physical table name, the storage model table supports infinite master-slave relation, and a corresponding physical table is generated according to the appointed physical table name.
Further, the timed acquisition is used for acquiring data by using CDC and ET L according to service data and index summarized data with low interactivity requirements;
the real-time acquisition is used for exchanging data by adopting a heterogeneous, multi-interface and stable message mechanism aiming at service data with high requirement on interaction urgency.
Further, the data element management includes standard code management and verification management.
A model-driven data integration middleware implementation method comprises the following steps:
s1, building a model driving data center;
s2, integrating data;
and S3, sharing data exchange.
Further, in step S1, the method includes:
s11, defining a corresponding information model including value domain codes, data elements and data set information through an integrated middleware configuration platform according to specific service requirements;
s12, creating a subject library according to business needs, generating a data storage model according to information model configuration, associating the data storage model with the subject library, generating a database table structure based on the storage model, completing creation of the database table, and establishing a complete data storage system, wherein the data storage system comprises constraint conditions of each data field, cleaning and conversion rules, whether encryption transmission and storage are needed or not can be set through a configuration platform.
Further, in step S2,
s21, the access side creates a related standard interface library at the exchange node according to the standard interface library script of the data integration platform;
s22, the access side loads the service data of the information system related to the exchange node to the standard interface library of the data integration platform according to the standard specification of the data integration platform; the whole process is divided into two stages, wherein the first stage is to import all historical data into a standard interface library of the data integration platform in batches, and the second stage is to import newly generated incremental data into the standard interface library of the data integration platform every day;
s23, performing data quality evaluation on the data in the standard interface library of the data integration platform through a data quality evaluation tool to ensure that the data quality meets the relevant requirements of the data integration platform;
s24, the access party calls the data integration platform data interface service to realize data uploading through the data integration platform data interface service client;
s25, the data integration platform evaluates the data quality of the data uploaded by the data access party, and uploads the data to the data center of the data integration platform under the condition that the data quality meets the project requirements after evaluation; and under the condition that the data quality does not meet the project requirements, feeding the data quality report back to a data uploading unit, requiring the data uploading unit to be modified, and uploading the data again after meeting the requirements of the data integration platform.
Further, in step S3,
s31, the data integration middleware generates data query interface service according to the data exchange and sharing interface specification configuration based on the information model and the storage model;
s32, the data exchange node is used for exchanging and sharing the interface service according to the data provided by the data integration platform and developing the client called by the related interface service according to the specification;
s33, calling information platform data exchange sharing service through the client, and finishing safety access control to the service and access authority authentication requested by the client by the information platform data exchange sharing service;
s34, the information platform data exchange sharing service realizes the query and acquisition of client request data by accessing the information platform data center, and assembles and returns the data in the format of interface specification;
and S35, the data exchange node acquires the information platform shared data returned by the information platform exchange for the service application of the data exchange node.
In the embodiment of the invention, the model-driven data integration middleware logically mainly comprises four major contents, and the integrated management of data integration is realized mainly through a model-driven design concept and a configuration management tool.
The first part is a data integration information model which is composed of value domain codes, data elements, data sets and other contents, and the information model can realize the structural management of data integration standards and can drive the data storage model to realize modeling.
The second part is a data integration storage and exchange model, which is composed of a subject library, a data storage model and a data exchange model, is realized based on an information model, mainly defines data storage rules and exchange rules, and supports the realization of data storage and exchange.
The third part is a data center, which is realized by a library-building script generated by a defined storage model, and realizes sharing and data exchange storage.
The fourth part is an integrated service, and the data storage and exchange service is configured and generated through a configuration management tool based on the set information model and the set storage model.
The model-driven data integration middleware technology can be structurally decomposed into two blocks of contents, namely a central-end service and a front gateway service.
The central service realizes the establishment of a data integration model, the formulation and release of standards and the unified management of data storage and exchange service, and realizes the business cooperation by relying on the standard service and the preposed gateway service to complete data integration.
The front gateway service is externally connected with a heterogeneous data source to be integrated, service cooperation is realized between the front gateway service and a central terminal service, standardized adaptation of a data structure and data content of heterogeneous data is realized by means of an information model standard downloaded from the central terminal, heterogeneous data acquisition is realized, and data integration is realized by calling data storage and exchange service released by the central terminal.
The main functional modules are as follows:
information model management module
The information model comprises value range codes, data elements and data set management related functions.
(1) And the module provides standardized codes, standardized conversion processing of non-standardized codes and mapping relation among related synonyms before information exchange and docking of different systems for internal and external application systems of the platform, provides uniform value domain codes for the whole platform, and provides a uniform standardized basis for data exchange and sharing, thereby being more beneficial to data acquisition and sharing of the system, statistics and analysis of shared data of platform users and data mining.
(2) And the data element management module comprises management of data element classification, data element details and the like (including whether standard code representation is required or not and whether verification is required or not). The data elements are the basis for unambiguous information exchange and data sharing between different business fields and are also the basic units for forming the data set.
(3) And managing the data set, wherein the information model mainly comprises data set classification, data set definition and data set detail. The data set is composed of data elements related to a specific business activity event and is used for describing a class of service activities or service events, an information model, an exchange specification, a data storage model and a storage specification of data exchange can be generated through the data set, the data set is a basis for data integration and exchange of a platform, and meanwhile, the data sets of different business domains, such as public health and basic medical treatment, are managed through data set classification.
Storage model management module
The data storage model comprises data source management, theme library management, data storage model management and the like.
(1) The data source management can mainly carry out operations such as adding, deleting, modifying, checking and the like on the data source, can maintain the pool attribute of the data source, and the system can generate a database connection word object for the dynamic query component of the platform according to the defined attribute. And meanwhile, database connection words can be provided for the data acquisition tool, and particularly, different subject libraries allowed by the system to be stored in different database instances or under different modes can also be defined herein.
(2) And managing a theme library, wherein the theme library is a classification established for all storage models in the same service domain according to different service domains, so that the management of system physical models, such as basic medical, public health, decision support and other theme libraries, is facilitated, and the system operates to assign different modes or database examples to each theme library and store the data of the theme library in a database.
(3) And managing a storage model, wherein the storage model table is used for selecting a corresponding data set from an information model resource library according to needs to be associated according to the information model in a specific service scene, and appointing a physical table name, the storage model table supports an infinite master-slave relationship, and a system generates a corresponding physical table according to the appointed physical table name.
(III) heterogeneous data integration module
The data integration service is divided into two modes, namely timing integration and real-time integration, the data collection is carried out on the service data and index summarized data with low interactive requirements at fixed time, CDC (data convergence) and ET L (discovery, understanding, modeling, extraction, cleaning, correction, standardization, isomerism, transmission, loading and registration) can be adopted for data collection, zero influence on hospital services can be realized, the data exchange is carried out on the service data with high interactive urgency requirements, such as infectious disease report, appointment registration APP application query, online instant query of inspection and inspection report and the like by adopting a heterogeneous, multi-interface and stable message mechanism.
The data collection service is a batch timing collection function of a platform for large data volume. The data acquisition and exchange layer essentially provides a uniform and reliable technical framework to realize the flexible exchange of the data acquired by the platform from the service system and the data between the platform data center and the service system. A certain relevance exists between the data acquisition and exchange layer and the business collaboration service layer, and a higher-level example in a data acquisition and exchange scenario is flow-driven data exchange, that is, data exchange is not only based on a simple link, but also needs to perform corresponding routing according to the content of some fields or approval opinions of some steps, based on a specific rule, and provides necessary services such as notification and the like in the exchange process. Such data collection and exchange need to be realized by some functions of process modeling. Data acquisition and exchange can be divided into three categories according to the technical implementation of the method: database-level data acquisition and exchange techniques, message-oriented data acquisition and exchange techniques, and service-oriented data acquisition and exchange techniques.
In order to relieve the transformation work of a data source information system, distribute the responsibility of both parties and promote project implementation, a front server scheme can be adopted in the initial-scale data acquisition process of a platform and a medical institution, so that the platform is required to provide a data acquisition gateway (Adapter) and an ET L data acquisition system to support data acquisition of heterogeneous systems of different hospitals and data loading to a data center, and data of information systems which are old, old and unattended and have poor use and maintenance capabilities of some medical institutions can be converted into data which accord with the data standard of the platform by matching with the data acquisition gateway and the ET L data acquisition system to be acquired to the platform data center.
(V) general exchange service management module
The method comprises the steps of abstracting a middle storage object (DataTable) of a platform on the basis of an information model and a storage model, storing transmitted data in a key value pair mode, and generating general adding, modifying, deleting and inquiring services according to a data set which needs to be stored and exchanged and a storage model structure.
The main process is as follows:
model-driven data center construction process
(1) The corresponding information model is defined through the integrated middleware configuration platform according to specific service requirements, the information model comprises value domain codes, data elements and data set information, and related contents can be realized through definition of one item of the platform and can also be imported in batches through an excel template provided by the platform.
(2) And creating a theme base according to the service requirement, generating a data storage model according to the information model configuration, and associating the data storage model with the theme base. Based on a storage model, a database table structure can be generated, the creation of the database table is completed, a complete data storage system is established, and the constraint conditions, cleaning and conversion rules of each data field, whether encryption transmission and storage are needed or not and the like can be set through a configuration platform.
(II) data integration flow
The process of realizing heterogeneous data integration based on the integration middleware driven by the model comprises the following steps:
(1) and the access party creates a related standard interface library at the switching node according to the standard interface library script of the data integration platform.
(2) And the access party loads the service data of the information system related to the switching node into a standard interface library of the data integration platform according to the standard specification related to the data integration platform. The whole process is divided into two stages, wherein the first stage is to import all historical data into the standard interface library of the data integration platform in batches, and the second stage is to import newly generated incremental data into the standard interface library of the data integration platform every day.
(3) The data imported into the standard interface library of the data integration platform by the access party needs to ensure the accuracy, integrity, relevance and timeliness of the data, and the data quality evaluation tool is used for evaluating the data quality of the data in the standard interface library of the data integration platform to ensure that the data quality meets the relevant requirements of the data integration platform.
(4) And the access party calls the data interface service of the data integration platform through the data interface service client of the data integration platform to realize data uploading.
(5) And the data integration platform carries out data quality evaluation on the data (stored in the pre-cache library) uploaded by the data access party, and uploads the data to the data center of the data integration platform under the condition that the data quality meets the project requirements after evaluation. And under the condition that the data quality does not meet the project requirements, feeding the data quality report back to a data uploading unit, requiring the data uploading unit to be modified, and uploading the data again after meeting the requirements of the data integration platform.
Data exchange sharing process:
(1) and the data integration middleware generates a data query interface service according to the data exchange and sharing interface specification configuration based on the information model and the storage model.
(2) And the data exchange node develops a client called by the related interface service according to the data exchange and sharing interface service and the specification provided by the data integration platform.
(3) And calling the information platform data exchange sharing service through the client, and finishing the safety access control of the service and the access authority authentication requested by the client by the information platform data exchange sharing service.
(4) The information platform data exchange sharing service realizes the query and acquisition of client request data by accessing an information platform data center, and the client request data is assembled and returned in a format with a standard interface.
(5) And the data exchange node acquires the information platform shared data exchanged and returned by the information platform and is used for the service application of the data exchange node.
In the specific embodiment of the present invention, the following embodiments are also included:
for example, the model-driven data integration middleware is actually a set of solution and a rapid development platform, and a set of configuration management tools and data exchange middleware provided by the platform can establish a path for integrating and exchanging the whole data without writing programs additionally, so that the model-driven data integration middleware has a strong service expansion capability, and the whole implementation process from the requirement proposition to the data integration, synchronization and exchange is as follows:
firstly, a corresponding information model, namely a data set, is defined through a configuration platform according to specific service requirements. The data set can be realized through the definition of a platform item, and can also be imported in batch through an excel template provided by the platform.
Secondly, a data storage model is configured and generated according to the data set model, a database table structure can be generated based on the storage model, the creation of the database table is completed, and a complete data storage system is established, wherein the complete data storage system comprises constraint conditions of each data field, cleaning and conversion rules, whether encryption transmission and storage are needed or not can be set through a configuration platform.
Then, based on the established data storage system, the front gateway service component can realize data acquisition and synchronous updating through an ET L tool, the task platform is mainly supported by three sets of tools, the first set of tool is a data acquisition process designer, the designer provides a visual interface to realize mapping between tables and conversion and synchronization of data, the second set of tool is a data synchronization service, the service can be used for carrying data of different service types through a general data transmission object which defines a key value pair, and remote synchronization of the data is realized in a remote service mode.
Finally, according to the data set, the generation of data exchange specifications, the definition of data interfaces and the release of services can be realized through configuration. The system administrator can configure the query data columns and the query conditions according to specific service requirements through the configuration platform. For the well-defined query service, the query service can be issued to an external system call in the form of a plurality of data transmission protocols and standard services through an enterprise service bus, and meanwhile, the data can be shown in some ways through configuration, some applications are constructed, and the like.
In other technical features of the embodiment, those skilled in the art can flexibly select and use the features according to actual situations to meet different specific actual requirements. However, it will be apparent to one of ordinary skill in the art that: it is not necessary to employ these specific details to practice the present invention. In other instances, well-known algorithms, methods or systems have not been described in detail so as not to obscure the present invention, and are within the scope of the present invention as defined by the claims.
For simplicity of explanation, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art will appreciate that the present application is not limited by the order of acts, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and elements referred to are not necessarily required in this application.
Those of skill in the art would appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The disclosed systems, modules, and methods may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units may be only one logical division, and there may be other divisions in actual implementation, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be referred to as an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may also be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It will be understood by those skilled in the art that all or part of the processes in the methods for implementing the embodiments described above can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a ROM, a RAM, etc.
The foregoing is illustrative of the preferred embodiments of this invention, and it is to be understood that the invention is not limited to the precise form disclosed herein and that various other combinations, modifications, and environments may be resorted to, falling within the scope of the concept as disclosed herein, either as described above or as apparent to those skilled in the relevant art. And that modifications and variations may be effected by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (9)
1. The model-driven data integration middleware is characterized by comprising an information model management module, a storage model management module, a heterogeneous data integration module and a general exchange service management module; the information model management module is used for value domain code management, data element management and data set management; the storage model management module is used for data source management, theme library management and storage model management; the heterogeneous data integration module is used for realizing two integration modes, namely timing integration and real-time integration; and the universal exchange service management module is used for abstracting a storage object in the middle of the platform on the basis of the information model and the storage model, storing the transmitted data in a key value pair mode, and generating universal adding, modifying, deleting and inquiring services according to the data set which needs to be stored and exchanged and the structure of the storage model.
2. A model-driven data integration middleware as claimed in claim 1,
the value domain code management provides standardized code and non-standardized code standardized conversion processing for the internal and external application systems of the platform, and the mapping relation between related synonyms before the information exchange and docking of different systems, provides uniform value domain codes for the whole platform, and provides uniform standardized basis for data exchange and sharing;
the data element management comprises data element classification management and data element detail management;
the data set management can generate an information model, an exchange specification, a data storage model and a storage specification of data exchange, and can manage data sets of different business domains by data set classification.
3. A model-driven data integration middleware as claimed in claim 1,
the data source management can perform the operations of adding, deleting, modifying and checking the data source, maintain the pool attribute of the data source and provide database connection words for a data acquisition tool;
the theme bank management is used for establishing classification for all storage models in the same service domain according to different service domains, so that management of a system physical model is facilitated, and meanwhile, the system operates to assign different modes or database examples for each theme bank and store the theme bank data in a database-by-database manner;
and the storage model management is used for managing a storage model table, the storage model table is used for selecting a corresponding data set from an information model resource library as required to be associated according to the information model in a specific service scene, and appointing a physical table name, the storage model table supports infinite master-slave relation, and a corresponding physical table is generated according to the appointed physical table name.
4. A model-driven data integration middleware as claimed in claim 1,
the timing acquisition is used for acquiring data by using CDC and ET L according to service data and index summarized data with low interactivity requirements;
the real-time acquisition is used for exchanging data by adopting a heterogeneous, multi-interface and stable message mechanism aiming at service data with high requirement on interaction urgency.
5. The model-driven data integration middleware of claim 2 wherein the data element management includes standard code management and verification management.
6. A method for implementing data integration middleware driven by a model is characterized by comprising the following steps:
s1, building a model driving data center;
s2, integrating data;
and S3, sharing data exchange.
7. The method for implementing the model-driven data integration middleware according to claim 6, wherein in step S1, the method comprises:
s11, defining a corresponding information model including value domain codes, data elements and data set information through an integrated middleware configuration platform according to specific service requirements;
s12, creating a subject library according to business needs, generating a data storage model according to information model configuration, associating the data storage model with the subject library, generating a database table structure based on the storage model, completing creation of the database table, and establishing a complete data storage system, wherein the data storage system comprises constraint conditions of each data field, cleaning and conversion rules, whether encryption transmission and storage are needed or not can be set through a configuration platform.
8. The method for implementing the model-driven data integration middleware of claim 6 wherein, in step S2,
s21, the access side creates a related standard interface library at the exchange node according to the standard interface library script of the data integration platform;
s22, the access side loads the service data of the information system related to the exchange node to the standard interface library of the data integration platform according to the standard specification of the data integration platform; the whole process is divided into two stages, wherein the first stage is to import all historical data into a standard interface library of the data integration platform in batches, and the second stage is to import newly generated incremental data into the standard interface library of the data integration platform every day;
s23, performing data quality evaluation on the data in the standard interface library of the data integration platform through a data quality evaluation tool to ensure that the data quality meets the relevant requirements of the data integration platform;
s24, the access party calls the data integration platform data interface service to realize data uploading through the data integration platform data interface service client;
s25, the data integration platform evaluates the data quality of the data uploaded by the data access party, and uploads the data to the data center of the data integration platform under the condition that the data quality meets the project requirements after evaluation; and under the condition that the data quality does not meet the project requirements, feeding the data quality report back to a data uploading unit, requiring the data uploading unit to be modified, and uploading the data again after meeting the requirements of the data integration platform.
9. The method for implementing the model-driven data integration middleware of claim 6 wherein, in step S3,
s31, the data integration middleware generates data query interface service according to the data exchange and sharing interface specification configuration based on the information model and the storage model;
s32, the data exchange node is used for exchanging and sharing the interface service according to the data provided by the data integration platform and developing the client called by the related interface service according to the specification;
s33, calling information platform data exchange sharing service through the client, and finishing safety access control to the service and access authority authentication requested by the client by the information platform data exchange sharing service;
s34, the information platform data exchange sharing service realizes the query and acquisition of client request data by accessing the information platform data center, and assembles and returns the data in the format of interface specification;
and S35, the data exchange node acquires the information platform shared data returned by the information platform exchange for the service application of the data exchange node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010145552.7A CN111400382A (en) | 2020-03-03 | 2020-03-03 | Model-driven data integration middleware and implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010145552.7A CN111400382A (en) | 2020-03-03 | 2020-03-03 | Model-driven data integration middleware and implementation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111400382A true CN111400382A (en) | 2020-07-10 |
Family
ID=71430487
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010145552.7A Pending CN111400382A (en) | 2020-03-03 | 2020-03-03 | Model-driven data integration middleware and implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400382A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112311893A (en) * | 2020-11-09 | 2021-02-02 | 成都市软件产业发展中心(成都信息化技术应用发展中心) | Cross-region, business and system data service middleware and data verification method |
| CN112395325A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data management method, system, terminal equipment and storage medium |
| CN112882848A (en) * | 2021-02-25 | 2021-06-01 | 武汉大学 | Basic middleware applied to network and distributed development |
| CN113010506A (en) * | 2021-03-11 | 2021-06-22 | 江苏省生态环境监控中心(江苏省环境信息中心) | Multi-source heterogeneous water environment big data management system |
| CN113051323A (en) * | 2021-03-11 | 2021-06-29 | 江苏省生态环境监控中心(江苏省环境信息中心) | Water environment big data exchange method |
| CN113312416A (en) * | 2021-05-20 | 2021-08-27 | 成都美尔贝科技股份有限公司 | Cross-data-center ETL tool |
| CN114265703A (en) * | 2022-03-02 | 2022-04-01 | 梯度云科技(北京)有限公司 | Cross-region computing power scheduling method, system and equipment for cloud server |
| CN114329076A (en) * | 2021-12-13 | 2022-04-12 | 珠海市新德汇信息技术有限公司 | Semi-structured data standard processing method, storage medium and equipment |
| CN114417448A (en) * | 2022-04-01 | 2022-04-29 | 创业慧康科技股份有限公司 | Data standard model design system and method based on Internet of things |
| CN114443005A (en) * | 2021-12-30 | 2022-05-06 | 中国航空研究院 | System integration method based on middleware |
| CN115718462A (en) * | 2022-07-19 | 2023-02-28 | 北京蓝晶微生物科技有限公司 | Method and device for equipment integration and automation control |
| CN116431616A (en) * | 2023-06-13 | 2023-07-14 | 深圳市诚识科技有限公司 | Big data model management system and method based on cloud computing |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2006026673A2 (en) * | 2004-08-31 | 2006-03-09 | Ascential Software Corporation | Architecture for enterprise data integration systems |
| CN103281368A (en) * | 2013-05-22 | 2013-09-04 | 河海大学 | Data sharing exchange model and method based on cloud computation |
-
2020
- 2020-03-03 CN CN202010145552.7A patent/CN111400382A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2006026673A2 (en) * | 2004-08-31 | 2006-03-09 | Ascential Software Corporation | Architecture for enterprise data integration systems |
| CN103281368A (en) * | 2013-05-22 | 2013-09-04 | 河海大学 | Data sharing exchange model and method based on cloud computation |
Non-Patent Citations (2)
| Title |
|---|
| 李曦柯;施荣华;许丹;陈练兵;潘航;: "基于全面互联互通和深度智能化建设医院信息集成平台" * |
| 杨立强;许丹;陈练兵;: "基层医疗卫生信息互联互通标准化建设探索" * |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112311893A (en) * | 2020-11-09 | 2021-02-02 | 成都市软件产业发展中心(成都信息化技术应用发展中心) | Cross-region, business and system data service middleware and data verification method |
| CN112395325A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data management method, system, terminal equipment and storage medium |
| CN112882848A (en) * | 2021-02-25 | 2021-06-01 | 武汉大学 | Basic middleware applied to network and distributed development |
| CN113010506A (en) * | 2021-03-11 | 2021-06-22 | 江苏省生态环境监控中心(江苏省环境信息中心) | Multi-source heterogeneous water environment big data management system |
| CN113051323A (en) * | 2021-03-11 | 2021-06-29 | 江苏省生态环境监控中心(江苏省环境信息中心) | Water environment big data exchange method |
| CN113051323B (en) * | 2021-03-11 | 2023-09-01 | 江苏省生态环境监控中心(江苏省环境信息中心) | Water environment big data exchange method |
| CN113010506B (en) * | 2021-03-11 | 2023-08-29 | 江苏省生态环境监控中心(江苏省环境信息中心) | Multi-source heterogeneous water environment big data management system |
| CN113312416B (en) * | 2021-05-20 | 2022-09-09 | 成都美尔贝科技股份有限公司 | Cross-data-center ETL tool |
| CN113312416A (en) * | 2021-05-20 | 2021-08-27 | 成都美尔贝科技股份有限公司 | Cross-data-center ETL tool |
| CN114329076A (en) * | 2021-12-13 | 2022-04-12 | 珠海市新德汇信息技术有限公司 | Semi-structured data standard processing method, storage medium and equipment |
| CN114443005A (en) * | 2021-12-30 | 2022-05-06 | 中国航空研究院 | System integration method based on middleware |
| CN114443005B (en) * | 2021-12-30 | 2024-04-30 | 中国航空研究院 | Middleware-based system integration method |
| CN114265703B (en) * | 2022-03-02 | 2022-05-20 | 梯度云科技(北京)有限公司 | Cross-region computing power scheduling method, system and equipment for cloud server |
| CN114265703A (en) * | 2022-03-02 | 2022-04-01 | 梯度云科技(北京)有限公司 | Cross-region computing power scheduling method, system and equipment for cloud server |
| CN114417448A (en) * | 2022-04-01 | 2022-04-29 | 创业慧康科技股份有限公司 | Data standard model design system and method based on Internet of things |
| CN115718462B (en) * | 2022-07-19 | 2023-09-29 | 北京蓝晶微生物科技有限公司 | Method and device for equipment integration and automatic control |
| CN115718462A (en) * | 2022-07-19 | 2023-02-28 | 北京蓝晶微生物科技有限公司 | Method and device for equipment integration and automation control |
| CN116431616A (en) * | 2023-06-13 | 2023-07-14 | 深圳市诚识科技有限公司 | Big data model management system and method based on cloud computing |
| CN116431616B (en) * | 2023-06-13 | 2023-09-01 | 深圳市诚识科技有限公司 | Big data model management system and method based on cloud computing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111400382A (en) | Model-driven data integration middleware and implementation method | |
| US8660987B2 (en) | Data cache techniques in support of synchronization of databases in a distributed environment | |
| CN107341643B (en) | Visual process display method and system | |
| US11971865B2 (en) | Intelligent model hierarchy for infrastructure modeling | |
| CN113849483A (en) | Real-time database system architecture for intelligent factory | |
| WO2021165524A1 (en) | Method and system for securing asset data in a computing environment | |
| CN112116516A (en) | Method and device for uniformly processing multiple types of government affairs items | |
| CN112988752A (en) | Resource management method, device, storage medium and electronic equipment | |
| CN110334147A (en) | A kind of method of data synchronization and device | |
| CN107896242B (en) | Service sharing method and device | |
| CN110019456B (en) | Data import method, device and system | |
| US20090210453A1 (en) | Service for verifying consistency of replicated data | |
| CN115543428A (en) | Simulated data generation method and device based on strategy template | |
| KR102298777B1 (en) | Integrated system including user interface | |
| CN114610803A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN101615272A (en) | A kind of approaches to IM and information management system | |
| CN114841678B (en) | Post data exchange method, data exchange system, server and storage medium | |
| RU2787261C1 (en) | System and method for engineering data management | |
| CN115758731B (en) | Advanced avionics architecture modeling tool | |
| TWI836916B (en) | Computer readable media, server and cross organizational project collaboration method the server implements | |
| Senchenko et al. | Architectural principles of common information space development for control of complex production processes | |
| CN114692590A (en) | Electronic document generation method and system | |
| CN117236865A (en) | Data management method and device, electronic equipment and storage medium | |
| CN115665241A (en) | Method for establishing system data integration interconnection general model | |
| CN118118493A (en) | Industrial Internet of things distributed data center table |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |