CN104699723A - Data exchange adapter and system and method for synchronizing data among heterogeneous systems - Google Patents
Data exchange adapter and system and method for synchronizing data among heterogeneous systems Download PDFInfo
- Publication number
- CN104699723A CN104699723A CN201310670951.5A CN201310670951A CN104699723A CN 104699723 A CN104699723 A CN 104699723A CN 201310670951 A CN201310670951 A CN 201310670951A CN 104699723 A CN104699723 A CN 104699723A
- Authority
- CN
- China
- Prior art keywords
- data
- cutting
- task
- plug
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data exchange adapter and a system and a method for synchronizing data among heterogeneous systems, and relates to the technical field of mass data processing. With a star-like mode design principle, a set of data exchange adapter device located between different types of databases or file systems is established, then the mass data are synchronized effectively among different heterogeneous systems, and the original net-shaped data synchronization mode formed by maintaining and managing a series of complicated data import and export tools is replaced. In addition, the embodiment of the invention provides a set of unified open framework to support secondary development, a DataIn plug-in and a DataOut plug-in provided by the scheme can be used or required DataIn plug-in and DataOut plug-in are self-defined, so that the task of synchronizing any mass data among the different types of databases or file systems is rapidly executed.
Description
Technical field
The present invention relates to mass data processing technical field, particularly data synchronous system and method between a kind of exchanges data adapter, heterogeneous system.
Background technology
In large-scale business production system application scenarios, often have database or the file system of number of different types, as Oracle, MySQL, SQL Server, Sybase, DB2, Informix, PostgreSQL, MongoDB, Redis, HDFS(Hadoop Distributed File System, Hadoop distributed file system) and other databases or FS(File System, file system) etc., and often need the operation such as importing or derivation carrying out mass data between which.A lot of towards the data exporting instrument of distinct type data-base or file system maturation at present, as MySQLDumper, OracleSQLLoader etc., but these instruments generally only provide derivation or the importing of data, and only support database or the file system of one or several particular type.
The follow-up increase along with operation system, the introduction in dissimilar storehouse and application, all can cause the linear increase to these data exporting instrument demands.Like this, we just need develop and safeguard a series of instrument, realize the data syn-chronization between disparate databases or file system, thus cause the data syn-chronization between different system to present a kind of reticulate texture, be illustrated in fig. 1 shown below, considerably increase the maintenance management difficulty of system.
In addition, these instruments are carry out exchange data with file mode (File) a bit, and some is with pipe method (Pipe) interim data, all to some extent for data syn-chronization brings overhead, cause efficiency variance very large.Simultaneously, a lot of instrument also effectively cannot meet data warehouse ETL(Extraction, Transformation & Loading, data are extracted, conversion and load) demand common in task, such as Time form transformation, special character conversion, code conversion etc.
Moreover sometimes, the situation of reality expects that we are within very short time interval, derive a data from a database, are loaded into multiple dissimilar database or file system simultaneously.
Summary of the invention
In view of the drawback of above-mentioned existence, the embodiment of the present invention provides a kind of data synchronous system and method between exchanges data adapter, heterogeneous system, realizes between different isomerization system, carrying out mass data synchronously, thus can effectively solve prior art defect.
The embodiment of the present invention have employed following technical scheme:
One embodiment of the invention provides a kind of exchanges data adapter, and described exchanges data adapter comprises:
Portable Batch System device, for carrying out configuration and the scheduling of data syn-chronization task between data system;
Data input plugin manager, comprise multiple data corresponding with several data system respectively and input plug-in unit, for obtaining the source data incoming task that described Portable Batch System device configures, call the data matched with described source data incoming task and input plug-in unit, obtain source data resource from source data system, and be loaded into data buffer storage manager;
Data buffer storage manager, for the source data Resourse Distribute storage space be loaded into for described data input plugin manager, and source data resource described in buffer memory;
Data output inserter manager, comprise multiple data output inserters corresponding with several data system respectively, for the operation plan according to described Portable Batch System device, trigger destination data and export task, and call the data output inserter exporting task with described destination data and match, from described data buffer storage manager, read corresponding source data resource, export destination data system to.
The data input plug-in unit of described data input plugin manager also comprises:
Source data resources shifting, for being set under starting state, according to the configuration of described Portable Batch System device, is the form that target data system is supported by the first data conversion module;
The data output inserter of described data output inserter manager also comprises:
Source data resources shifting, for being set under starting state, according to the configuration of described Portable Batch System device, is the form that target data system is supported by the second data conversion module.
Described Portable Batch System device comprises:
Task scheduling modules one to one, for carrying out configuration and the scheduling of a source data system and another destination data data among systems synchronous task;
One-to-many task scheduling modules, for carrying out configuration and the scheduling of a source data system and multiple destination data data among systems synchronous task;
Many-one task scheduling modules, for carrying out configuration and the scheduling of multiple source data system and a destination data data among systems synchronous task; And
Multi-to-multi task scheduling modules, for carrying out configuration and the scheduling of multiple source data system and multiple destination data data among systems synchronous task.
Described Portable Batch System device also comprises:
The cutting of source data resource for receiving the first cutting configuration information, if need carry out cutting to source data resource, is then multiple subtasks that can be concurrent according to described first cutting configuration information by incoming task cutting module; And/or
Output task cutting module, for receiving the second cutting configuration information, if cutting need be carried out to the data resource exporting destination data system to, be then multiple subtasks that can be concurrent according to described second cutting configuration information by the data resource cutting exporting destination data system to.
Described data input plugin manager also comprises:
Data input plug-in card configuration module, inputs plug-in unit for increasing the data corresponding with new data system according to increasing demand, and inputs plug-in unit according to the deletion demand deletion data corresponding with legacy data system;
Described data output inserter manager also comprises:
Data output inserter configuration module, for increasing the data output inserter corresponding with new data system according to increasing demand, and deletes the data output inserter corresponding with legacy data system according to the demand of deletion;
Described system also comprises:
Log manager, for recording the running log information of described Portable Batch System device, data input plugin manager, data buffer storage manager and data output inserter manager, to carry out tracking investigation when going wrong.
In addition, the embodiment of the present invention additionally provides data synchronous system between a kind of heterogeneous system, comprises several data system and exchanges data adapter described above;
The data output inserter that each described data system inputs corresponding types in plugin manager respectively data with described data input corresponding types in plug-in unit and described data output inserter manager is connected.
In addition, the embodiment of the present invention additionally provides method of data synchronization between a kind of heterogeneous system, and when carrying out data syn-chronization operation between data system, described method comprises:
The exchanges data adapter be connected with each data system respectively carries out configuration and the scheduling of data syn-chronization task between data system;
According to configuration and schedule information, obtain source data resource from source data system, and be buffered in described exchanges data adapter;
Trigger destination data according to operation plan and export task, read corresponding source data resource, and export destination data system to.
Describedly also to comprise after source data system obtains source data resource:
According to configuration information, be the form that target data system is supported by source data resources shifting; Or
Also comprise after the source data resource that described reading is corresponding:
According to configuration information, be the form that target data system is supported by source data resources shifting.
Describedly also comprising after source data system obtains source data resource: receive the first cutting configuration information, if need carry out cutting to source data resource, is then multiple subtasks that can be concurrent according to described first cutting configuration information by the cutting of source data resource; Described being buffered in described exchanges data adapter is specially: obtain after buffer memory cutting described multiple can be concurrent subtask.
And/or, described export destination data system to before also comprise: receive the second cutting configuration information, if cutting need be carried out to the data resource exporting destination data system to, be then multiple subtasks that can be concurrent according to described second cutting configuration information by the data resource cutting exporting destination data system to;
The described destination data system that exports to is specially: what cutting obtained described multiplely can export destination data system in concurrent subtask, realizes data syn-chronization.
Described method also comprises:
Record the running log information of described exchanges data adapter, to carry out tracking investigation when going wrong.
Visible, data synchronous system and method between a kind of exchanges data adapter that the embodiment of the present invention provides, heterogeneous system, by this based on Star Schema design philosophy, build a set of exchanges data adapter apparatus be located between distinct type data-base or file system, effectively can realize carrying out mass data between different isomerization system synchronously, replace the mesh data method of synchronization that a series of data exporting instruments of original maintenance management complexity are formed.
In addition, embodiments provide a set of unified open frame, support secondary development, the DataIn(data input that this programme can be used to provide) plug-in unit and DataOut(data export) plug-in unit or customed DataIn and DataOut plug-in unit needed for oneself, and realize secondary development based on the plugin manager that the embodiment of the present invention provides, load different classes of DataIn and DataOut plug-in unit by easy configuration, thus realize any mass data synchronous task between dissimilar database or file system fast.
Accompanying drawing explanation
Fig. 1 be in prior art between different pieces of information system based on data syn-chronization relation formed reticulate texture schematic diagram;
A kind of exchanges data adapter structure block diagram that Fig. 2 provides for the embodiment of the present invention;
A kind of exchanges data adapter structure schematic diagram that Fig. 3 provides for the embodiment of the present invention;
Data synchronous system structural representation between a kind of heterogeneous system that Fig. 4 provides for the embodiment of the present invention;
Method of data synchronization process flow diagram between a kind of heterogeneous system that Fig. 5 provides for the embodiment of the present invention;
Fig. 6 for the embodiment of the present invention provide a kind of from Oracle data system carry out one to one synchrodata to the exchanges data adapter inner processing procedure schematic diagram of MySQL data system;
What Fig. 7 provided for the embodiment of the present invention a kind ofly carries out a pair two synchrodatas exchanges data adapter inner processing procedure schematic diagram to MySQL data system and SQL Server data system from Oracle data system;
What Fig. 8 provided for the embodiment of the present invention a kind ofly carries out two to the exchanges data adapter inner processing procedure schematic diagram of a synchrodata to SQL Server data system from Oracle data system and MySQL data system;
What Fig. 9 provided for the embodiment of the present invention a kind ofly carries out two to the exchanges data adapter inner processing procedure schematic diagram of two synchrodatas to SQL Server data system and Sybase data system from Oracle data system and MySQL data system.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly, below in conjunction with accompanying drawing, embodiment of the present invention is described further in detail.
See Fig. 2, the embodiment of the present invention provides a kind of exchanges data adapter (DXA, Data eXchangeAdapter), specifically comprises:
Portable Batch System device 201, for carrying out configuration and the scheduling of data syn-chronization task between data system.
Data input plugin manager 202, comprise multiple data corresponding with several data system respectively and input plug-in unit, for obtaining the source data incoming task that described Portable Batch System device 201 configures, call the data matched with described source data incoming task and input plug-in unit, obtain source data resource from source data system, and be loaded into data buffer storage manager 203.
Data buffer storage manager 203, for the source data Resourse Distribute storage space be loaded into for described data input plugin manager 202, and source data resource described in buffer memory.
Data output inserter manager 204, comprise multiple data output inserters corresponding with several data system respectively, for the operation plan according to described Portable Batch System device 201, trigger destination data and export task, and call the data output inserter exporting task with described destination data and match, from described data buffer storage manager 203, read corresponding source data resource, export destination data system to.
It should be noted that, different pieces of information system specifically comprises distinct type data-base and file system etc.The data base management system (DBMS) adopted towards different business application demand or distributed file system, the oracle database adopted as used for products information management, HDFS of adopting My SQL database and Based on Distributed cloud computing towards sequence information management and adopt etc.
Preferably, the data input DataIn plug-in unit of described data input plugin manager also comprises:
Source data resources shifting, for being set under starting state, according to the configuration of described Portable Batch System device, is the form that target data system is supported by the first data conversion module;
The data of described data output inserter manager export DataOut plug-in unit and also comprise:
Source data resources shifting, for being set under starting state, according to the configuration of described Portable Batch System device, is the form that target data system is supported by the second data conversion module.
That is, be built-in with algorithm in advance in each data input DataIn plug-in unit, data mode data system corresponding for this plug-in unit can supported is converted into the data mode that other different pieces of information system is supported.In like manner, each data export in DataOut plug-in unit and are also built-in with algorithm, the data mode that other different pieces of information system is supported can be converted into the data mode that data system corresponding to this plug-in unit is supported.In specific implementation, according to configuration, the first data conversion module or the second data conversion module can be opened, namely according to configuration, can select to carry out data mode conversion in data input plug-in unit, or carry out data mode conversion in data output inserter.
Wherein, DataIn plug-in unit can be connected multiple of the same type or dissimilar Database Systems or file system with the plug-in unit configured in DataOut plugin manager, but the database of each type or file system must have plug-in unit corresponding with it to connect with it.
It should be noted that, the package types that the embodiment of the present invention can be supported has multiple, namely can configure and load on data and input or output corresponding plug-in unit in plugin manager, comprise OracleDataIn, MySQLDataIn, SQLServerDataIn, SybaseDataIn, IBMDB2DataIn, InformixDataIn, PostgreSQLDataIn, HDFSDataIn, StreamDataIn, HTTPDataIn, MongoDBDataIn, RedisDataIn and OracleDataOut, MySQLDataOut, SQLServerDataOut, SybaseDataOut, IBMDB2DataOut, InformixDataOut, PostgreSQLDataOut, HDFSDataOut, StreamDataOut, HTTPDataOut, MongoDBDataOut, plug-in unit that RedisDataOut etc. and the follow-up open frame mentioned based on this programme are developed etc.
The data that table 1 below and table 2 list current exchanges data adapter apparatus DXA can be provided input DataIn plug-in unit and data export DataOut plug-in unit:
Table 1 data input DataIn plug-in unit
| Plugin name | Plug-in unit explanation |
| OracleDataIn | Support to obtain data from oracle database system |
| MySQLDataIn | Support to obtain data from MySQL database system |
| SQLServerDataIn | Support to obtain data from MS SQL Server Database Systems |
| SybaseDataIn | Support to obtain data from sybase database system |
| IBMDB2DataIn | Support to obtain data from IBM DB2 Database Systems |
| InformixDataIn | Support to obtain data from informix database system |
| PostgreSQLDataIn | Support to obtain data from PostgreSQL Database Systems |
| HDFSDataIn | Support to obtain data from HDFS file system |
| StreamDataIn | Support to obtain data from Stream stream |
| HTTPDataIn | Support to obtain data from http URL |
| MongoDBDataIn | Support to obtain data from MongoDB Database Systems |
| RedisDataIn | Support to obtain data from Redis Database Systems |
Table 2 data export DataOut plug-in unit
| Plugin name | Plug-in unit explanation |
| OracleDataOut | Support to export data to oracle database system |
| MySQLDataOut | Support to export data to MySQL database system |
| SQLServerDataOut | Support to export data to MS SQL Server Database Systems |
| SybaseDataOut | Support to export data to sybase database system |
| IBMDB2DataOut | Support to export data to IBM DB2 Database Systems |
| InformixDataOut | Support to export data to informix database system |
| PostgreSQLDataOut | Support to export data to PostgreSQL Database Systems |
| HDFSDataOut | Support to export data to HDFS file system |
| StreamDataOut | Support to export data to Stream stream |
| HTTPDataOut | Support to export data to http URL |
| MongoDBDataOut | Support to export data to MongoDB Database Systems |
| RedisDataOut | Support to export data to Redis Database Systems |
It should be noted that in addition, the plug-in unit configured in DataIn with DataOut plugin manager can be connected multiple of the same type or dissimilar Database Systems or file system, but the database of each type or file system must have plug-in unit corresponding with it to connect with it.
Concrete, described Portable Batch System device comprises:
Task scheduling modules one to one, for carrying out configuration and the scheduling of a source data system and another destination data data among systems synchronous task.
One-to-many task scheduling modules, for carrying out configuration and the scheduling of a source data system and multiple destination data data among systems synchronous task.
Many-one task scheduling modules, for carrying out configuration and the scheduling of multiple source data system and a destination data data among systems synchronous task.
And multi-to-multi task scheduling modules, for carrying out configuration and the scheduling of multiple source data system and multiple destination data data among systems synchronous task.
That is, the data syn-chronization task between different isomerization system exists one to one, the relation of one-to-many, many-one, multi-to-multi.
Preferably, described Portable Batch System device also comprises:
The cutting of source data resource for receiving the first cutting configuration information, if need carry out cutting to source data resource, is then multiple subtasks that can be concurrent according to described first cutting configuration information by incoming task cutting module.
And/or, output task cutting module, for receiving the second cutting configuration information, if cutting need be carried out to the data resource exporting destination data system to, be then multiple subtasks that can be concurrent according to described second cutting configuration information by the data resource cutting exporting destination data system to.
That is, when needs carry out inputing or outputing task, can be configured cutting situation, time such as from oracle database to MySQL database synchrodata, oracle database needs synchronous data to relate to 50 tables, and MySQL database is according to business actual demand, needs the data resource of reception to be deposited in 100 storehouse tables, then preferred, in incoming task cutting module and the cutting configuration that task cutting module carries out data syn-chronization task respectively can be exported.At incoming task cutting block configuration cutting configuration information for 50 involved by oracle database are table according to the cutting of practical business demand to be multiple source data resource tasks to be read; At output task cutting block configuration cutting configuration information for 100 involved by MySQL database are table according to the cutting of practical business demand to be multiple destination data resource tasks to be written; And then carry out data syn-chronization.In certain practical application, task cutting can be carried out at input end to reading task according to business actual demand, by its cutting be multiple can the little task of parallel processing, also can according to business actual demand output terminal to read from storage area task cutting be multiple can the little task of parallel processing, namely slicing operation can carry out at input end, also can carry out at output terminal, input end and output terminal determine whether to need to carry out cutting according to practical business demand, like this, after task cutting, data syn-chronization is simplified greatly, and is more convenient to management.
It should be noted that, the data syn-chronization task segmentation principle mentioned in the embodiment of the present invention, be not one to one, the principle one_to_one corresponding of to be namely not task based access control be one-to-many or many-one or multi-to-multi carries out cutting, but carries out cutting according to business actual demand completely.
The embodiment of the present invention is based on Star Schema thought, a set of exchanges data adapter apparatus be located between different isomerization system of design (is abbreviated as DXA, Data eXchange Adapter), this device adopts the open architecture of framework (Framework)+plug-in unit (Plugin) to build, its middle frame passes through data buffering, thread pool encapsulates, the technology such as flow control realize data syn-chronization, and provide simple interface and plug-in unit mutual, plug-in unit is then for accessing dissimilar database or file system provides realization, different classes of DataIn and DataOut plug-in unit is loaded by easy configuration, thus can realize fast carrying out arbitrary data exchange between dissimilar database or file system.
In addition, what the embodiment of the present invention proposed carries out the synchronous implementation of mass data between heterogeneous system, and design corresponding exchanges data adapter apparatus, solve outside the mesh data method of synchronization of operation management complexity, as preferably, in the embodiment of the present invention, exchanges data adapter is on the unified open frame basis provided, and can support secondary development, according to business demand, can customed required DataIn and DataOut(data input and data output) plug-in unit; Based on the plugin manager that this device provides, load different classes of DataIn and DataOut plug-in unit by easy configuration, any mass data that can realize fast between dissimilar database or file system is synchronous.
Concrete, described data input plugin manager also comprises:
Data input plug-in card configuration module, inputs plug-in unit for increasing the data corresponding with new data system according to increasing demand, and inputs plug-in unit according to the deletion demand deletion data corresponding with legacy data system.
And described data output inserter manager also comprises:
Data output inserter configuration module, for increasing the data output inserter corresponding with new data system according to increasing demand, and deletes the data output inserter corresponding with legacy data system according to the demand of deletion.
Especially, based on the open frame of the support secondary development that this programme is mentioned, according to business actual demand, the DataIn plug-in unit and the DataOut plug-in unit that meet certain database system or file system can be developed more.
Further, described system also comprises:
Log manager, for recording the running log information of described Portable Batch System device, data input plugin manager, data buffer storage manager and data output inserter manager, to carry out tracking investigation when going wrong.
That is, log manager can be called by other managers of DXA, the daily record of generation system ruuning situation, so that platform carries out tracking investigation in operational process during produced problem.
Preferably, the ephemeral data storage space that data buffer storage manager generates can be the mode based on internal memory, also can be the mode of physically based deformation disk storage, and this point is configured according to the promptness of service synchronization data and the magnitude of data to be determined.
Visible, a kind of exchanges data adapter that the embodiment of the present invention provides, by this based on Star Schema design philosophy, build a set of exchanges data adapter apparatus be located between distinct type data-base or file system, effectively can realize carrying out mass data between different isomerization system synchronously, replace the mesh data method of synchronization that a series of data exporting instruments of original maintenance management complexity are formed.
In addition, embodiments provide a set of unified open frame, support secondary development, the DataIn(data input that this programme can be used to provide) plug-in unit and DataOut(data export) plug-in unit or customed DataIn and DataOut plug-in unit needed for oneself, and realize secondary development based on the plugin manager that the embodiment of the present invention provides, load different classes of DataIn and DataOut plug-in unit by easy configuration, thus realize any mass data synchronous task between dissimilar database or file system fast.
See Fig. 3, a kind of exchanges data adapter instance that the embodiment of the present invention provides, figure 3 illustrates the inside composition structure of exchanges data adapter apparatus DXA and the incidence relation of correspondence, relate generally to Portable Batch System device, data input plugin manager, data buffer storage manager, data output inserter manager and platform running log manager etc.
Be described as follows:
portable Batch System device
Comprise source data incoming task scheduler handler and destination data output Portable Batch System device, between responsible dissimilar system, carry out the management such as the configuration of data syn-chronization task, cutting and scheduling.Needs illustrate a bit, and the data syn-chronization task between different isomerization system exists one to one, the relation of one-to-many, many-one, multi-to-multi.
data input plugin manager
Be responsible for the configuration management of all DataIn plug-in units that DXA supports, as OracleDataIn, MySQLDataIn etc., and according to the source data incoming task that Portable Batch System device configures, adopt data input plug-in unit corresponding with it, relevant source data resource is loaded into the data space that data buffer storage manager distributes temporarily, reads for data output inserter relevant in data output inserter manager.
data buffer storage manager
Be responsible for data input and data and export the management such as the generation of ephemeral data storage space when carrying out exchanges data, maintenance and destruction.
data output inserter manager
Be responsible for the configuration management of all DataOut plug-in units that DXA supports, as OracleDataOut, MySQLDataOut etc., and export task according to the destination data that Portable Batch System device configures, adopt data output inserter corresponding with it, source data resource in the ephemeral data storage space that reading data buffer storage manager distributes, and output to destination data system.
platform running log manager
Called by other managers of DXA, the daily record of generation system ruuning situation, so that platform carries out tracking investigation in operational process during produced problem.
See Fig. 4, the embodiment of the present invention additionally provides data synchronous system between a kind of heterogeneous system, specifically comprises several data system, and the exchanges data adapter as described in above-mentioned Fig. 1-Fig. 2.
Wherein, the data output inserter that the data that each described data system inputs corresponding types in plugin manager respectively with described data input corresponding types in plug-in unit and described data output inserter manager is connected.
Be described as follows:
(1) different pieces of information system, comprises distinct type data-base or file system
The data base management system (DBMS) adopted towards different business application demand or distributed file system, the oracle database adopted as used for products information management, HDFS of adopting MySQL database and Based on Distributed cloud computing towards sequence information management and adopt etc.
In the diagram, different pieces of information system is illustrated for Oracle, MySQL, SQLServer, Sybase, IBM DB2, Informix, HDFS, Other DB & FS respectively.
(2) exchanges data adapter apparatus
According to practical business demand, by configuring and loading different classes of DataIn and DataOut plug-in unit, realize carrying out exchanges data between dissimilar database or file system.
It should be noted that, the modules in present system embodiment or the principle of work of unit and processing procedure see the associated description in exchanges data adapter embodiments shown in above-mentioned Fig. 1-Fig. 3, can repeat no more herein.
See Fig. 5, embodiments provide method of data synchronization between a kind of heterogeneous system, specifically comprise the steps:
S501: when carrying out data syn-chronization operation between data system, the exchanges data adapter be connected with each data system respectively carries out configuration and the scheduling of data syn-chronization task between data system.
S502: according to configuration and schedule information, obtains source data resource from source data system, and is buffered in described exchanges data adapter.
S503: trigger destination data according to operation plan and export task, read corresponding source data resource, and export destination data system to.
It should be noted that, data syn-chronization task between different isomerization system exists one to one, the relation of one-to-many, many-one, multi-to-multi.
Wherein, describedly also to comprise after source data system obtains source data resource: according to configuration information, be the form that target data system is supported by source data resources shifting,
Or also comprising after the source data resource that described reading is corresponding: according to configuration information, is the form that target data system is supported by source data resources shifting.
That is, be built-in with algorithm in advance in each data input DataIn plug-in unit, data mode data system corresponding for this plug-in unit can supported is converted into the data mode that other different pieces of information system is supported.In like manner, each data export in DataOut plug-in unit and are also built-in with algorithm, the data mode that other different pieces of information system is supported can be converted into the data mode that data system corresponding to this plug-in unit is supported.In specific implementation, according to configuration, the first data conversion module or the second data conversion module can be opened, namely according to configuration, can select to carry out data mode conversion in data input plug-in unit, or carry out data mode conversion in data output inserter.
Wherein, DataIn plug-in unit can be connected multiple of the same type or dissimilar Database Systems or file system with the plug-in unit configured in DataOut plugin manager, but the database of each type or file system must have plug-in unit corresponding with it to connect with it.
Preferably, describedly also comprising after source data system obtains source data resource: receive the first cutting configuration information, if need carry out cutting to source data resource, is then multiple subtasks that can be concurrent according to described first cutting configuration information by the cutting of source data resource.
Accordingly, be buffered in described in described exchanges data adapter and be specially: obtain after buffer memory cutting described multiple can be concurrent subtask.
And/or, described export destination data system to before also comprise: receive the second cutting configuration information, if cutting need be carried out to the data resource exporting destination data system to, be then multiple subtasks that can be concurrent according to described second cutting configuration information by the data resource cutting exporting destination data system to.
Accordingly, export destination data system to and be specially described in: what cutting obtained described multiplely can export destination data system in concurrent subtask, realizes data syn-chronization.
That is, when needs carry out inputing or outputing task, can be configured cutting situation, the i.e. cutting of input/output task, carry out cutting according to the size actual conditions of data volume to be synchronized, according to the configuration for cutting in exchanges data adapter, carry out concrete slicing operation at input end or output terminal cutting.
It should be noted that, the data syn-chronization task segmentation principle mentioned in the embodiment of the present invention, be not one to one, the principle one_to_one corresponding of to be namely not task based access control be one-to-many or many-one or multi-to-multi carries out cutting, but carries out cutting according to business actual demand completely.
The embodiment of the present invention is based on Star Schema thought, a set of exchanges data adapter apparatus be located between different isomerization system of design (is abbreviated as DXA, Data eXchange Adapter), this device adopts the open architecture of framework (Framework)+plug-in unit (Plugin) to build, its middle frame passes through data buffering, thread pool encapsulates, the technology such as flow control realize data syn-chronization, and provide simple interface and plug-in unit mutual, plug-in unit is then for accessing dissimilar database or file system provides realization, different classes of DataIn and DataOut plug-in unit is loaded by easy configuration, thus can realize fast carrying out arbitrary data exchange between dissimilar database or file system.
In addition, what the embodiment of the present invention proposed carries out the synchronous implementation of mass data between heterogeneous system, and design corresponding exchanges data adapter apparatus, solve outside the mesh data method of synchronization of operation management complexity, as preferably, in the embodiment of the present invention, exchanges data adapter is on the unified open frame basis provided, and can support secondary development, according to business demand, can customed required DataIn and DataOut(data input and data output) plug-in unit; Based on the plugin manager that this device provides, load different classes of DataIn and DataOut plug-in unit by easy configuration, any mass data that can realize fast between dissimilar database or file system is synchronous.
Also namely concrete, the method that the embodiment of the present invention provides also comprises: increase the data corresponding with new data system according to increasing demand and input plug-in unit, and inputs plug-in unit according to the deletion demand deletion data corresponding with legacy data system.
Especially, based on the open frame of the support secondary development that this programme is mentioned, according to business actual demand, the DataIn plug-in unit and the DataOut plug-in unit that meet certain database system or file system can be developed more.
Further, the method that the embodiment of the present invention provides also comprises:
Record the running log information of described exchanges data adapter, to carry out tracking investigation when going wrong.That is, log manager can be called by other managers of DXA, the daily record of generation system ruuning situation, so that platform carries out tracking investigation in operational process during produced problem.
It should be noted that, the processing procedure of each step in the inventive method embodiment and principle of work see each module in exchanges data adapter embodiments shown in Fig. 1-Fig. 3 and system shown in Figure 4 embodiment or the principle of work of unit and the associated description of processing procedure, can repeat no more herein.
Visible, method of data synchronization between a kind of heterogeneous system that the embodiment of the present invention provides, by this based on Star Schema design philosophy, build a set of exchanges data adapter apparatus be located between distinct type data-base or file system, effectively can realize carrying out mass data between different isomerization system synchronously, replace the mesh data method of synchronization that a series of data exporting instruments of original maintenance management complexity are formed.
In addition, embodiments provide a set of unified open frame, support secondary development, the DataIn(data input that this programme can be used to provide) plug-in unit and DataOut(data export) plug-in unit or customed DataIn and DataOut plug-in unit needed for oneself, and realize secondary development based on the plugin manager that the embodiment of the present invention provides, load different classes of DataIn and DataOut plug-in unit by easy configuration, thus realize any mass data synchronous task between dissimilar database or file system fast.
See Fig. 6, illustrate that the embodiment of the present invention provides a kind of from Oracle data system carry out one to one synchrodata to the exchanges data adapter inner processing procedure of MySQL data system.
Be synchronized to the implementation procedure of MySQL system (data destination) DXA from Oracle system (data source) one to one, be described as follows:
In source data incoming task scheduler handler, configure corresponding Oracle Data entry tasks, if task is larger, according to the actual requirements, can be multiple subtasks that can be concurrent by task cutting.
Exporting in destination data the MySQL data that in Portable Batch System device, configuration is corresponding with step one and export task, if task is larger, can be multiple subtasks that can be concurrent by task cutting according to the actual requirements.
Configure in data input plugin manager and load OracleDataIn plug-in unit, and connecting with oracle database system.
Configure in data output inserter manager and load MySQLDataOut plug-in unit, and connecting with MySQL database.
Data syn-chronization scheduling time according to Portable Batch System device arranges, OracleDataIn plug-in unit notification data cache manager generates the temporary memory space of exchanges data, and is loaded into the data space that data buffer storage manager generates temporarily after being responsible for that the source data in oracle database is carried out suitable computing.
OracleDataIn plug-in unit completes after source data is loaded into ephemeral data storage space, according to the relevant configuration information in Portable Batch System device, can trigger or delay the source data triggered in MySQLDataOut plug-in unit reading temporary memory space in time, and after carrying out suitable computing, output to MySQL database system.
After OracleDataIn plug-in unit and MySQLDataOut plug-in unit complete exchanges data task, data buffer storage manager can be destroyed ephemeral data storage space and discharge.
See Fig. 7, what illustrate that the embodiment of the present invention provides a kind ofly carries out a pair two synchrodatas exchanges data adapter inner processing procedure to MySQL data system and SQL Server data system from Oracle data system.
Carry out from Oracle system (data source) implementation procedure being synchronized to MySQL system and SQL Server system (data destination) DXA for a pair two, be described as follows:
In source data incoming task scheduler handler, configure corresponding Oracle Data entry tasks, if task is larger, according to the actual requirements, can be multiple subtasks that can be concurrent by task cutting.
Exporting in destination data MySQL and the SQLServer data that in Portable Batch System device, configuration is corresponding with step one and export task, if task is larger, can be multiple subtasks that can be concurrent by task cutting according to the actual requirements.
Configure in data input plugin manager and load OracleDataIn plug-in unit, and connecting with oracle database system.
Configure in data output inserter manager and load MySQLDataOut and SQLServerDataOut plug-in unit, and connecting respectively with MySQL and SQL Server database.
Data syn-chronization scheduling time according to Portable Batch System device arranges, OracleDataIn plug-in unit notification data cache manager generates the temporary memory space of exchanges data, and is loaded into the data space that data buffer storage manager generates temporarily after being responsible for that the source data in oracle database is carried out suitable computing.
OracleDataIn plug-in unit completes after source data is loaded into ephemeral data storage space, according to the relevant configuration information in Portable Batch System device, can trigger in time or delay and trigger MySQLDataOut plug-in unit and SQLServerDataOut plug-in unit and read source data in temporary memory space respectively, and after carrying out suitable computing, output to MySQL and SQL Server Database Systems.
After OracleDataIn plug-in unit and MySQLDataOut plug-in unit and SQLServerDataOut plug-in unit complete exchanges data task, data buffer storage manager can be destroyed ephemeral data storage space and discharge.
See Fig. 8, illustrate that a kind of that the embodiment of the present invention provides carries out two to the exchanges data adapter inner processing procedure of a synchrodata to SQL Server data system from Oracle data system and MySQL data system.
Carry out from Oracle and MySQL system (data source) two to be synchronized to SQL Server system (data destination) DXA implementation procedure to one, be described as follows:
In source data incoming task scheduler handler, configure corresponding Oracle and MySQL Data entry tasks, if task is larger, according to the actual requirements, can be multiple subtasks that can be concurrent by task cutting.
Exporting in destination data the SQL Server data that in Portable Batch System device, configuration is corresponding with step one and export task, if task is larger, can be multiple subtasks that can be concurrent by task cutting according to the actual requirements.
Configure in data input plugin manager and load OracleDataIn plug-in unit and MySQLDataIn plug-in unit, and connecting respectively with Oracle and MySQL database system.
Configure in data output inserter manager and load SQLServerDataOut plug-in unit, and connecting with SQLServer Database Systems.
Data syn-chronization scheduling time according to Portable Batch System device arranges, OracleDataIn plug-in unit and MySQLDataIn plug-in unit respectively notification data cache manager generate the temporary memory space of exchanges data, and are loaded into the data space that data buffer storage manager generates temporarily after being responsible for that the source data in Oracle and MySQL database is carried out suitable computing.
OracleDataIn plug-in unit and MySQLDataIn plug-in unit complete after source data is loaded into ephemeral data storage space, according to the relevant configuration information in Portable Batch System device, can trigger or delay the source data triggered in SQLServerDataOut plug-in unit reading temporary memory space in time, and after carrying out suitable computing, output to SQL Server Database Systems.
After OracleDataIn, MySQLDataIn plug-in unit and SQLServerDataOut plug-in unit complete exchanges data task, data buffer storage manager can be destroyed ephemeral data storage space and discharge.
See Fig. 9, what illustrate that the embodiment of the present invention provides a kind ofly carries out two to the exchanges data adapter inner processing procedure of two synchrodatas to SQL Server data system and Sybase data system from Oracle data system and MySQL data system.
Carry out from Oracle and MySQL system (data source) two to be synchronized to SQL Server and Sybase system (data destination) DXA implementation procedure to two, be described as follows:
In source data incoming task scheduler handler, configure corresponding Oracle and MySQL Data entry tasks, if task is larger, according to the actual requirements, can be multiple subtasks that can be concurrent by task cutting.
Exporting the configuration SQL Server corresponding with step one and Sybase data in Portable Batch System device in destination data and export task, if task is larger, can be multiple subtasks that can be concurrent by task cutting according to the actual requirements.
Configure in data input plugin manager and load OracleDataIn plug-in unit and MySQLDataIn plug-in unit, and connecting respectively with Oracle and MySQL database system.
Configure in data output inserter manager and load SQLServerDataOut plug-in unit and SybaseDataOut plug-in unit, and connecting respectively with SQL Server and sybase database system.
Data syn-chronization scheduling time according to Portable Batch System device arranges, OracleDataIn plug-in unit and MySQLDataIn plug-in unit respectively notification data cache manager generate the temporary memory space of exchanges data, and are loaded into the data space that data buffer storage manager generates temporarily after being responsible for that the source data in Oracle and MySQL database is carried out suitable computing.
OracleDataIn plug-in unit and MySQLDataIn plug-in unit complete after source data is loaded into ephemeral data storage space, according to the relevant configuration information in Portable Batch System device, can trigger or delay the source data in triggering SQLServerDataOut plug-in unit corresponding with it and SybaseDataOut plug-in unit reading temporary memory space in time, and after carrying out suitable computing, output to SQL Server and sybase database system.
After OracleDataIn, MySQLDataIn plug-in unit and SQLServerDataOut, SybaseDataOut plug-in unit complete exchanges data task, data buffer storage manager can be destroyed ephemeral data storage space and discharge.
For the ease of the technical scheme of the clear description embodiment of the present invention, in inventive embodiment, have employed the printed words such as " first ", " second " to distinguish the substantially identical identical entry of function and efficacy or similar item, it will be appreciated by those skilled in the art that the printed words such as " first ", " second " do not limit quantity and execution order.
One of ordinary skill in the art will appreciate that, the all or part of step realized in above-described embodiment method is that the hardware that can carry out instruction relevant by program has come, described program can be stored in a computer read/write memory medium, this program is when performing, comprise the steps: (step of method), described storage medium, as: ROM/RAM, magnetic disc, CD etc.
The foregoing is only preferred embodiment of the present invention, be not intended to limit protection scope of the present invention.All any amendments done within the spirit and principles in the present invention, equivalent replacement, improvement etc., be all included in protection scope of the present invention.
Claims (10)
1. an exchanges data adapter, is characterized in that, described exchanges data adapter comprises:
Portable Batch System device, for carrying out configuration and the scheduling of data syn-chronization task between data system;
Data input plugin manager, comprise multiple data corresponding with several data system respectively and input plug-in unit, for obtaining the source data incoming task that described Portable Batch System device configures, call the data matched with described source data incoming task and input plug-in unit, obtain source data resource from source data system, and be loaded into data buffer storage manager;
Data buffer storage manager, for the source data Resourse Distribute storage space be loaded into for described data input plugin manager, and source data resource described in buffer memory;
Data output inserter manager, comprise multiple data output inserters corresponding with several data system respectively, for the operation plan according to described Portable Batch System device, trigger destination data and export task, and call the data output inserter exporting task with described destination data and match, from described data buffer storage manager, read corresponding source data resource, export destination data system to.
2. exchanges data adapter according to claim 1, is characterized in that, the data input plug-in unit of described data input plugin manager also comprises:
Source data resources shifting, for being set under starting state, according to the configuration of described Portable Batch System device, is the form that target data system is supported by the first data conversion module;
The data output inserter of described data output inserter manager also comprises:
Source data resources shifting, for being set under starting state, according to the configuration of described Portable Batch System device, is the form that target data system is supported by the second data conversion module.
3. exchanges data adapter according to claim 1, is characterized in that, described Portable Batch System device comprises:
Task scheduling modules one to one, for carrying out configuration and the scheduling of a source data system and another destination data data among systems synchronous task;
One-to-many task scheduling modules, for carrying out configuration and the scheduling of a source data system and multiple destination data data among systems synchronous task;
Many-one task scheduling modules, for carrying out configuration and the scheduling of multiple source data system and a destination data data among systems synchronous task; And
Multi-to-multi task scheduling modules, for carrying out configuration and the scheduling of multiple source data system and multiple destination data data among systems synchronous task.
4. exchanges data adapter according to claim 1, is characterized in that, described Portable Batch System device also comprises:
The cutting of source data resource for receiving the first cutting configuration information, if need carry out cutting to source data resource, is then multiple subtasks that can be concurrent according to described first cutting configuration information by incoming task cutting module; And/or
Output task cutting module, for receiving the second cutting configuration information, if cutting need be carried out to the data resource exporting destination data system to, be then multiple subtasks that can be concurrent according to described second cutting configuration information by the data resource cutting exporting destination data system to.
5. exchanges data adapter according to claim 1, is characterized in that, described data input plugin manager also comprises:
Data input plug-in card configuration module, inputs plug-in unit for increasing the data corresponding with new data system according to increasing demand, and inputs plug-in unit according to the deletion demand deletion data corresponding with legacy data system;
Described data output inserter manager also comprises:
Data output inserter configuration module, for increasing the data output inserter corresponding with new data system according to increasing demand, and deletes the data output inserter corresponding with legacy data system according to the demand of deletion.
6. exchanges data adapter according to claim 1, is characterized in that, described system also comprises:
Log manager, for recording the running log information of described Portable Batch System device, data input plugin manager, data buffer storage manager and data output inserter manager, to carry out tracking investigation when going wrong.
7. a data synchronous system between heterogeneous system, is characterized in that, comprises several data system and the exchanges data adapter as described in any one of claim 1-6;
The data output inserter that each described data system inputs corresponding types in plugin manager respectively data with described data input corresponding types in plug-in unit and described data output inserter manager is connected.
8. a method of data synchronization between heterogeneous system, is characterized in that, when carrying out data syn-chronization operation between data system, described method comprises:
The exchanges data adapter be connected with each data system respectively carries out configuration and the scheduling of data syn-chronization task between data system;
According to configuration and schedule information, obtain source data resource from source data system, and be buffered in described exchanges data adapter;
Trigger destination data according to operation plan and export task, read corresponding source data resource, and export destination data system to.
9. method according to claim 8, is characterized in that, describedly also comprises after source data system obtains source data resource:
According to configuration information, be the form that target data system is supported by source data resources shifting; Or
Also comprise after the source data resource that described reading is corresponding:
According to configuration information, be the form that target data system is supported by source data resources shifting.
10. method according to claim 8, it is characterized in that, describedly also to comprise after source data system obtains source data resource: receive the first cutting configuration information, if cutting need be carried out to source data resource, be then multiple subtasks that can be concurrent according to described first cutting configuration information by the cutting of source data resource; Described being buffered in described exchanges data adapter is specially: obtain after buffer memory cutting described multiple can be concurrent subtask; And/or
Described export destination data system to before also comprise: receive the second cutting configuration information, if cutting need be carried out to the data resource exporting destination data system to, be then multiple subtasks that can be concurrent according to described second cutting configuration information by the data resource cutting exporting destination data system to; The described destination data system that exports to is specially: what cutting obtained described multiplely can export destination data system in concurrent subtask, realizes data syn-chronization;
Described method also comprises:
Record the running log information of described exchanges data adapter, to carry out tracking investigation when going wrong.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310670951.5A CN104699723B (en) | 2013-12-10 | 2013-12-10 | Data synchronous system and method between data exchange adapter, heterogeneous system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310670951.5A CN104699723B (en) | 2013-12-10 | 2013-12-10 | Data synchronous system and method between data exchange adapter, heterogeneous system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104699723A true CN104699723A (en) | 2015-06-10 |
| CN104699723B CN104699723B (en) | 2018-10-19 |
Family
ID=53346855
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310670951.5A Active CN104699723B (en) | 2013-12-10 | 2013-12-10 | Data synchronous system and method between data exchange adapter, heterogeneous system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104699723B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN106250444A (en) * | 2016-07-27 | 2016-12-21 | 北京集奥聚合科技有限公司 | The real-time Input System of a kind of heterogeneous data source and method |
| CN106469152A (en) * | 2015-08-14 | 2017-03-01 | 阿里巴巴集团控股有限公司 | A kind of document handling method based on ETL and system |
| CN106682036A (en) * | 2015-11-11 | 2017-05-17 | 上海汽车集团股份有限公司 | Data exchange system and exchange method thereof |
| CN109344140A (en) * | 2018-08-14 | 2019-02-15 | 北京奇虎科技有限公司 | Data access method, device, electronic equipment and computer storage medium |
| CN109582722A (en) * | 2018-11-30 | 2019-04-05 | 珠海市新德汇信息技术有限公司 | Public security resource data service system |
| CN109828947A (en) * | 2019-01-17 | 2019-05-31 | 东莞众家联供应链服务有限公司 | A kind of active isomeric data collector |
| CN109992544A (en) * | 2019-04-02 | 2019-07-09 | 镇江市第四人民医院（镇江市妇幼保健院） | A kind of intermediate mimicry system and its mimicry method |
| WO2019153553A1 (en) * | 2018-02-12 | 2019-08-15 | 平安科技（深圳）有限公司 | Cross wide area network data return method and apparatus, computer device, and storage medium |
| CN110851513A (en) * | 2019-10-16 | 2020-02-28 | 中盈优创资讯科技有限公司 | Multi-source heterogeneous data reading method and device based on Spark calculation engine |
| CN111538772A (en) * | 2020-04-14 | 2020-08-14 | 北京宝兰德软件股份有限公司 | Data exchange processing method and device, electronic equipment and storage medium |
| CN112134927A (en) * | 2020-08-26 | 2020-12-25 | 国电南瑞科技股份有限公司 | Power grid data exchange platform and data exchange method based on plug-in mode |
| CN112565042A (en) * | 2020-12-24 | 2021-03-26 | 航天科工网络信息发展有限公司 | Method for exchanging star-structured data |
| CN117931767A (en) * | 2023-12-15 | 2024-04-26 | 北京宇信科技集团股份有限公司 | Database migration platform and database migration method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6016501A (en) * | 1998-03-18 | 2000-01-18 | Bmc Software | Enterprise data movement system and method which performs data load and changed data propagation operations |
| CN1717674A (en) * | 2001-12-12 | 2006-01-04 | 索尼电子有限公司 | Transforming multimedia data for delivery to multiple heterogeneous devices |
| CN101243443A (en) * | 2005-08-24 | 2008-08-13 | 微软公司 | Security in peer to peer synchronization applications |
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN103020147A (en) * | 2012-11-22 | 2013-04-03 | 用友软件股份有限公司 | Data integration system and data integration method |
-
2013
- 2013-12-10 CN CN201310670951.5A patent/CN104699723B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6016501A (en) * | 1998-03-18 | 2000-01-18 | Bmc Software | Enterprise data movement system and method which performs data load and changed data propagation operations |
| CN1717674A (en) * | 2001-12-12 | 2006-01-04 | 索尼电子有限公司 | Transforming multimedia data for delivery to multiple heterogeneous devices |
| CN101243443A (en) * | 2005-08-24 | 2008-08-13 | 微软公司 | Security in peer to peer synchronization applications |
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN103020147A (en) * | 2012-11-22 | 2013-04-03 | 用友软件股份有限公司 | Data integration system and data integration method |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106469152A (en) * | 2015-08-14 | 2017-03-01 | 阿里巴巴集团控股有限公司 | A kind of document handling method based on ETL and system |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN106682036A (en) * | 2015-11-11 | 2017-05-17 | 上海汽车集团股份有限公司 | Data exchange system and exchange method thereof |
| CN106250444A (en) * | 2016-07-27 | 2016-12-21 | 北京集奥聚合科技有限公司 | The real-time Input System of a kind of heterogeneous data source and method |
| WO2019153553A1 (en) * | 2018-02-12 | 2019-08-15 | 平安科技（深圳）有限公司 | Cross wide area network data return method and apparatus, computer device, and storage medium |
| CN109344140A (en) * | 2018-08-14 | 2019-02-15 | 北京奇虎科技有限公司 | Data access method, device, electronic equipment and computer storage medium |
| CN109582722A (en) * | 2018-11-30 | 2019-04-05 | 珠海市新德汇信息技术有限公司 | Public security resource data service system |
| CN109828947A (en) * | 2019-01-17 | 2019-05-31 | 东莞众家联供应链服务有限公司 | A kind of active isomeric data collector |
| CN109992544A (en) * | 2019-04-02 | 2019-07-09 | 镇江市第四人民医院（镇江市妇幼保健院） | A kind of intermediate mimicry system and its mimicry method |
| CN110851513A (en) * | 2019-10-16 | 2020-02-28 | 中盈优创资讯科技有限公司 | Multi-source heterogeneous data reading method and device based on Spark calculation engine |
| CN110851513B (en) * | 2019-10-16 | 2021-06-08 | 中盈优创资讯科技有限公司 | Multi-source heterogeneous data reading method and device based on Spark calculation engine |
| CN111538772A (en) * | 2020-04-14 | 2020-08-14 | 北京宝兰德软件股份有限公司 | Data exchange processing method and device, electronic equipment and storage medium |
| CN112134927A (en) * | 2020-08-26 | 2020-12-25 | 国电南瑞科技股份有限公司 | Power grid data exchange platform and data exchange method based on plug-in mode |
| CN112565042A (en) * | 2020-12-24 | 2021-03-26 | 航天科工网络信息发展有限公司 | Method for exchanging star-structured data |
| CN117931767A (en) * | 2023-12-15 | 2024-04-26 | 北京宇信科技集团股份有限公司 | Database migration platform and database migration method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104699723B (en) | 2018-10-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104699723A (en) | Data exchange adapter and system and method for synchronizing data among heterogeneous systems | |
| US10069916B2 (en) | System and method for transparent context aware filtering of data requests | |
| US9336288B2 (en) | Workflow controller compatibility | |
| CN103379159B (en) | A kind of method that distributed Web station data synchronizes | |
| CN108681569B (en) | Automatic data analysis system and method thereof | |
| CN110674154B (en) | Spark-based method for inserting, updating and deleting data in Hive | |
| CN105205154B (en) | Data migration method and device | |
| US20130227194A1 (en) | Active non-volatile memory post-processing | |
| CN103927338A (en) | Log information storage processing method and log information storage processing device | |
| CN110019267A (en) | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium | |
| US20150074115A1 (en) | Distributed storage of data | |
| CN104317928A (en) | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database | |
| CN107766147A (en) | Distributed data analysis task scheduling system | |
| CN110471754A (en) | Method for exhibiting data, device, equipment and storage medium in job scheduling | |
| CN114417408B (en) | Data processing method, device, equipment and storage medium | |
| CN105701190A (en) | Data synchronizing method and device | |
| CN107832323B (en) | Distributed realization system and method based on graph database | |
| CN104915902A (en) | Cloud platform based implementation method for take-out order online delivery | |
| CN112988702A (en) | Heterogeneous data source real-time data transmission method and system, storage medium and terminal | |
| CN106156047A (en) | A kind of SNAPSHOT INFO processing method and processing device | |
| CN105516284A (en) | Clustered database distributed storage method and device | |
| CN115033646A (en) | Method for constructing real-time warehouse system based on Flink and Doris | |
| CN111367953A (en) | Streaming processing method and device for information data | |
| Park et al. | An implementation of a high throughput data ingestion system for machine logs in manufacturing industry | |
| CN104407811B (en) | A kind of merging I/O device based on cloud computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 818, 8 / F, 34 Haidian Street, Haidian District, Beijing 100080 Patentee after: BEIJING ULTRAPOWER SOFTWARE Co.,Ltd. Address before: 100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building 6 storey block A Room 601 Patentee before: BEIJING ULTRAPOWER SOFTWARE Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |