A kind of loose coupling data integration synchronization realizing method based on enterprise-level application scene
Technical field
The present invention relates to database technical field, is a kind of loose coupling number based on enterprise-level application scene specifically
According to integrated synchronous implementation method.
Background technology
Data synchronization technology in computer realm is always one, and constantly in the problem of solution, technical implementation way is various,
Realization principle is also different.The target of data synchronization technology is to maintain that data are consistent or data result is consistent, multiple business
Systems face has to develop a set of new data syn-chronization when integrated to be integrated for multiservice system.Traditional data syn-chronization is done
Method(As shown in Figure 1):1)Hibernate blockers set out data syn-chronization, by data push to cache pool;2)Task
Manager scheduling threads read data in cache pool;3)Data are subjected to secondary encapsulation, add data type;4)Task
Manager dispatches adapter according to data type and carries out data distribution, completes data syn-chronization.The problem of existing be:Iteration cycle
It hurry up, either management end or client, any demand change will account for synchrodata, and product versatility is not strong;
Push mode and destination address are write extremely in a program, and coupling is high, integrate very flexible;Relation management can not be synchronized, together
Step process black box, effective management of data can not be realized, loss of data may be caused.Close coupling framework essence is one
Client/Server models.Client computer originates request to server, and server receives, and is responded according to request, then feeds back
To client computer.With the continuous expansion of Laxcus cluster scales, visit capacity is continuously increased, especially data amount of calculation, calculating
Time, close coupling framework can't bear the heavy load gradually, problem be present into after the growth of multiple:1. it can not support to calculate industry on a large scale
Business.Because big data business is universal to computer resource accounting very big, cause multi-task parallel limited in one's ability.2. computer load
It is uncontrollable.In other words, as computer can not control overloading, and it is very big to hardware injury to overload, and this can be serious
Reduce computer stable operation ability and service life.3. management difficulty is big during tasks carrying.Task is in the process of implementation
Not by management and control.4. pair network resource consumption is big.Between data are sent and data return, it is idle to have very big one section for simultaneously operating
, this idle occupancy is the significant wastage to Internet resources.5. security control dynamics is poor.Because server is directly exposed to visitor
Family machine, easily triggers attack.6. the degree of association is too high between program code, it is unfavorable for modularized processing.It is existing more than 7.
It is deteriorated as ultimately resulting in the stability of a system.
The content of the invention
It is an object of the invention to provide a kind of synchronous realization side of loose coupling data integration based on enterprise-level application scene
Method, for solve in the prior art the coupling height of data, data integration very flexible, relation management can not be synchronized, can
The problem of loss of data can be caused.
In order to achieve the above object, the present invention is achieved through the following technical solutions:
A kind of loose coupling data integration synchronization realizing method based on enterprise-level application scene, comprises the following steps:
S1)In the self-defined API of application layer, connector is configured, the connector connects Connector, the pipe of the Connector
Reason mode is divided into local management and service end unified management, and the way to manage can be selected voluntarily under different application scenarios
Select;
S2)IDC Server can carry out the lifecycle management of data synchronization process and checking, manage and tracing for data, institute
State IDC Server to interact with application layer, and realize the synchronization of data, and transfer data to resource layer;
S3)The resource layer receives the data of IDC Server inputs, and by the routing table matched, different data sources is passed through into road
By being forwarded.
Data syn-chronization based on more adapters integrates(IDC)Compared with other data syn-chronization integration modes, most importantly
A variety of data syn-chronization mode standards are proposed, are realized farthest without between modification integration mode and multiple synchronization mode
Free switching, the lifecycle management of process and synchrodata is synchronized, solve the various Data Migrations in implementation process
With the integrated application scenarios problem such as multi-data source.A variety of data syn-chronization modes, it is synchronous using the data integration of loose coupling, have more
High flexibility, suitable for Data Migration and plurality of application scenes.
Preferably, the way to manage of the Connector uses the API and SAPI of standard, the standard of the API and SAPI
Including actively and passively two kinds of push mode, for realizing the free switching of different application scene under a variety of data formats.
Define API and SAPI standard, in accordance with nominally, can under a variety of data formats free switching(File,
Database, catalogue etc.), to be flexibly applicable various scenes.And API and SAPI design is comprising actively and passively two kinds push moulds
Formula, synchronized relation can carry out online dynamic configuration in the configuration page, the ease for use and versatility with height.
Preferably, manager, the manager include built in the IDC Server:
Connector Manager:The API of offer standard, and synchronize the online management of configuration and the scheduling of internal thread;
Account Manager:Authentication API and data traffic control API are provided, carry out data decryptor, current limliting or out of service
Operation;
Strategy Manager:Policy service API is provided, the scheduling for IDC Server provides support;
SyncData Manager:Data conversion is provided and encapsulates external API;
SAPI Manager:High-level interface between application program and speech engine is provided, is externally sudden and violent based on multi-data source
The API standard interface of dew.
A variety of managers realize the lifecycle management to synchronizing process built in IDC Server, and data can be checked, managed
Reason and retrospect.Connector Manager provide the API of standard, and synchronize the online management and internal thread of configuration
Scheduling, it is ensured that IDC Server Effec-tive Function.Account Manager provide authentication API and data traffic control API, entirely
Cheng Jinhang data decryptors, current limliting or out of service can be carried out according to strategy if necessary.Strategy Manager provide strategy clothes
Be engaged in API, and the scheduling for IDC Server provides support.SyncData Manager provide data conversion and encapsulate external API,
It is IDC Server API standard formulation reference, predefined data structure can be converted into by the service using the standard.
SAPI Manager are one of interfaces of API standard externally exposed based on multi-data source.
Preferably, the IDC Server also include MSG Server and AUDIT Server, the MSG Server use
In receiving, transmit and forward the message of the manager, and it is transferred to the safety that AUDIT Server carry out data transmission sending
Audit.
MSG Server use new message mechanism, are mainly used in solving the skills such as messenger service is stuck and message channel blocks
Art problem.Account:To guarantee data security, data channel is encrypted using AES, and using section technology to institute
There is request to be intercepted and authenticated, wherein in being realized in section, used new algorithm to realize keyword generation and parsing, improved
The operational efficiency of section.Strategy:In the technology of strategy use, new change has been carried out based on original policy techniques mode
Enter, new algorithm adjustment has all been carried out to the structure of tactful generating mode and strategy, has allowed strategy matching speed to improve 50%.
AUDIT:Depth calculation technology is added on original audit message basis, is refined by depth calculation, allows Audit data total amount
30% is have dropped, and data validity then greatly promotes.IDC Manager:To realize synchronous efficient management, in synchronizing process
It is middle to have used black/white list technology, the caching technology based on keyword, the time of synchronizing process is greatly reduced, is improved same
Walk efficiency
Preferably, the step S2)Middle IDC Server realize that the specific steps of data syn-chronization include:
S2.1)Login management by synchronization center configuration administration page, configuration data synchronized relation strategy, generation strategy mark and strategy
Content;
S2.2)Strategy is checked, using strategy mark rather than query strategy content, health status inspection is carried out using quick echo plex mode
Look into, by limiting time, whether returning result confirms path situation;
S2.3)Operation system or other data sources initiate data synchronization request, or are started according to the IDC Server of configuration
SAPI carries out data acquisition to target source;
S2.4)The personalized Connector customized according to the integrated protocol between operation system configures to read, and calls IDC
Server API carries out data-pushing;
S2.5)IDC Server proceed by distribution, by according to the data syn-chronization routing policy being pre-configured with, obtain opposite end
Connector list, the Connector of the list match is called to carry out data transmission;
S2.6)After Connector matches target source address, start to call corresponding method of data synchronization strategy progress data same
Step, while message informing is sent to IDC Server, represent that data syn-chronization has been completed, Connector calls IDC API, adopts
Make data syn-chronization with thread pool treatment technology, the processing inside IDC to message has carried out differentiated control:Level one data is behavior
Data are directly put in storage, and secondary data is that status data is directly put in storage, and three DBMSs are the data operated, call depth calculation side
Method calculates data syn-chronization.
Connector can promoter and processor each other, terminate to be used as closed loop using Connector processing.It is but all same
Step is not handled by Connector and is used as trigger mechanism, to avoid the formation of endless loop.IDC Server scheduling processes are configurable
, and corresponding strategy is can configure to control synchronous Thread Count or I/O expenses and historical data processing etc..
Independently extension needs to realize IDC SAPI standards Connector, main to include 3 aspects:
Configuration Interface:Configuration interface is realized, mainly connects configuration information;
Connector Interface:Connecting interface is realized, mainly specifies connection type;
Operation Interfaces:Computing interface is realized, is mainly used in the editor of specific synchrodata.
Preferably, the step S2.2)In strategy check include:
A. determination strategy whether there is;
B. whether determination strategy state enables;
C. whether destination address can use in determination strategy;
D. opposite end Connector patterns are judged.
To ensure that data synchronize according to established rule, it is necessary to carry out policy resolution and carry out corresponding operation to ensure
Strategy operation.
The present invention compared with prior art, has advantages below and beneficial effect:
(1)Data syn-chronization of the present invention is realized with operation system and decoupled, and can realize data syn-chronization by modes such as interface or configurations, is had
There is the business irrelevant character of height.IDC Server make transparent procedures to the lifecycle management of synchronizing process and synchrodata
Change, configurationization, at the same make data from generation to can inquire about, manage and trace in the Life cycle of same the end of the step.
(2)The present invention proposes data standard for conventional and unconventional mode synchrodata, in the situation in accordance with standard
The achievable data syn-chronization without modification integrates down, services the elastic realization principle of inside and can tackle a variety of data syn-chronization scenes,
In critical field, including asymmetric agreement and multi-protocols scene, can freely it switch over.
Brief description of the drawings
Fig. 1 is the system principle diagram of data syn-chronization in the prior art;
Fig. 2 is the system principle diagram of the present invention;
Fig. 3 is the system principle diagram that IDC Server realize data syn-chronization decoupling.
Embodiment
Give to being designed to proprietary vocabulary herein and explain such as before being described in detail to the specific embodiment of the present invention first
Under:
Section(Aspect):It is exactly functional realization altogether in fact.Such as daily record section, authority section, affairs section.In reality
Be typically the common java class that a shared function of storage is realized in, why can by AOP container identifications into section, be
Specified in configuration;
The concept of white list is corresponding with " blacklist ".After blacklist enables, the user of blacklist is formulated into(Or IP address,
IP bags, mail, virus etc.)It can not pass through.If set up white list, the user in white list(Or IP address, IP bags,
Mail etc.)Can be preferentially by the way that spam rejection will not be treated as, and security and agility all greatly improve;
Thread Pool Technology:One thread pool includes following four element:Thread pool manager(ThreadPool):With
In creating and manage thread pool, including thread pool is created, destroying threads pond, add new task;Worker thread(PoolWorker):
Thread pool thread, it is waited in no task, the execution task that can be circulated;Task interface(Task):Each
The interface that task must be realized, for the execution of worker thread scheduler task, it mainly defines the entrance of task, tasks carrying
Round-off work after complete, the execution state of task etc.;Task queue(taskQueue):For depositing not handling for task, carry
For a kind of buffering;
Authenticate (authentication)Refer to the right for verifying whether user possesses access system.
The present invention is described in further detail with reference to embodiment, but the implementation of the present invention is not limited to this.
Embodiment 1:
With reference to shown in accompanying drawing 2, a kind of loose coupling data integration synchronization realizing method based on enterprise-level application scene, including it is following
Step:
S1)In the self-defined API of application layer, connector is configured, the connector connects Connector, the pipe of the Connector
Reason mode is divided into local management and service end unified management, and the way to manage can be selected voluntarily under different application scenarios
Select;
S2)IDC Server can carry out the lifecycle management of data synchronization process and checking, manage and tracing for data,
The IDC Server interact with application layer, and realize the synchronization of data, and transfer data to resource layer;
S3)The resource layer receives the data of IDC Server inputs, and by the routing table matched, different data sources is passed through into road
By being forwarded.
Data syn-chronization based on more adapters integrates(IDC)Compared with other data syn-chronization integration modes, most importantly
A variety of data syn-chronization mode standards are proposed, are realized farthest without between modification integration mode and multiple synchronization mode
Free switching, the lifecycle management of process and synchrodata is synchronized, solve the various Data Migrations in implementation process
With the integrated application scenarios problem such as multi-data source.A variety of data syn-chronization modes, it is synchronous using the data integration of loose coupling, have more
High flexibility, suitable for Data Migration and plurality of application scenes.
Embodiment 2:
On the basis of embodiment 1, with reference to shown in accompanying drawing 2, the way to manage of the Connector using standard API and
SAPI, the API and SAPI standard include actively and passively two kinds of push mode, different under a variety of data formats for realizing
The free switching of application scenarios.
Define API and SAPI standard, in accordance with nominally, can under a variety of data formats free switching(File,
Database, catalogue etc.), to be flexibly applicable various scenes.And API and SAPI design is comprising actively and passively two kinds push moulds
Formula, synchronized relation can carry out online dynamic configuration in the configuration page, the ease for use and versatility with height.
Embodiment 3:
On the basis of embodiment 2, with reference to shown in accompanying drawing 2 and Fig. 3, manager built in the IDC Server, the manager
Including:
Connector Manager:The API of offer standard, and synchronize the online management of configuration and the scheduling of internal thread;
Account Manager:Authentication API and data traffic control API are provided, carry out data decryptor, current limliting or out of service
Operation;
Strategy Manager:Policy service API is provided, the scheduling for IDC Server provides support;
SyncData Manager:Data conversion is provided and encapsulates external API;
SAPI Manager:High-level interface between application program and speech engine is provided, is externally sudden and violent based on multi-data source
The API standard interface of dew.
A variety of managers realize the lifecycle management to synchronizing process built in IDC Server, and data can be checked, managed
Reason and retrospect.Connector Manager provide the API of standard, and synchronize the online management and internal thread of configuration
Scheduling, it is ensured that IDC Server Effec-tive Function.Account Manager provide authentication API and data traffic control API, entirely
Cheng Jinhang data decryptors, current limliting or out of service can be carried out according to strategy if necessary.Strategy Manager provide strategy clothes
Be engaged in API, and the scheduling for IDC Server provides support.SyncData Manager provide data conversion and encapsulate external API,
It is IDC Server API standard formulation reference, predefined data structure can be converted into by the service using the standard.
SAPI Manager are one of interfaces of API standard externally exposed based on multi-data source.
Preferably, the IDC Server also include MSG Server and AUDIT Server, the MSG Server use
In receiving, transmit and forward the message of the manager, and it is transferred to the safety that AUDIT Server carry out data transmission sending
Audit.
MSG Server use new message mechanism, are mainly used in solving the skills such as messenger service is stuck and message channel blocks
Art problem.Account:To guarantee data security, data channel is encrypted using AES, and using section technology to institute
There is request to be intercepted and authenticated, wherein in being realized in section, used new algorithm to realize keyword generation and parsing, improved
The operational efficiency of section.Strategy:In the technology of strategy use, new change has been carried out based on original policy techniques mode
Enter, new algorithm adjustment has all been carried out to the structure of tactful generating mode and strategy, has allowed strategy matching speed to improve 50%.
AUDIT:Depth calculation technology is added on original audit message basis, is refined by depth calculation, allows Audit data total amount
30% is have dropped, and data validity then greatly promotes.IDC Manager:To realize synchronous efficient management, in synchronizing process
It is middle to have used black/white list technology, the caching technology based on keyword, the time of synchronizing process is greatly reduced, is improved same
Walk efficiency.
Embodiment 4:
On the basis of embodiment 3, with reference to shown in accompanying drawing 2 and Fig. 3, the step S2)Middle IDC Server realize data syn-chronization
Specific steps include:
S2.1)Login management by synchronization center configuration administration page, configuration data synchronized relation strategy, generation strategy mark and strategy
Content;
S2.2)Strategy is checked, using strategy mark rather than query strategy content, health status inspection is carried out using quick echo plex mode
Look into, by limiting time, whether returning result confirms path situation;
S2.3)Operation system or other data sources initiate data synchronization request, or are started according to the IDC Server of configuration
SAPI carries out data acquisition to target source;
S2.4)The personalized Connector customized according to the integrated protocol between operation system configures to read, and calls IDC
Server API carries out data-pushing;
S2.5)IDC Server proceed by distribution, by according to the data syn-chronization routing policy being pre-configured with, obtain opposite end
Connector list, the Connector of the list match is called to carry out data transmission;
S2.6)After Connector matches target source address, start to call corresponding method of data synchronization strategy progress data same
Step, while message informing is sent to IDC Server, represent that data syn-chronization has been completed, Connector calls IDC API, adopts
Make data syn-chronization with thread pool treatment technology, the processing inside IDC to message has carried out differentiated control:Level one data is behavior
Data are directly put in storage, and secondary data is that status data is directly put in storage, and three DBMSs are the data operated, call depth calculation side
Method calculates data syn-chronization.
Connector can promoter and processor each other, terminate to be used as closed loop using Connector processing.It is but all same
Step is not handled by Connector and is used as trigger mechanism, to avoid the formation of endless loop.IDC Server scheduling processes are configurable
, and corresponding strategy is can configure to control synchronous Thread Count or I/O expenses and historical data processing etc..
Independently extension needs to realize IDC SAPI standards Connector, main to include 3 aspects:
Configuration Interface:Configuration interface is realized, mainly connects configuration information;
Connector Interface:Connecting interface is realized, mainly specifies connection type;
Operation Interfaces:Computing interface is realized, is mainly used in the editor of specific synchrodata.
Preferably, the step S2.2)In strategy check include:
A. determination strategy whether there is;
B. whether determination strategy state enables;
C. whether destination address can use in determination strategy;
D. opposite end Connector patterns are judged.
To ensure that data synchronize according to established rule, it is necessary to carry out policy resolution and carry out corresponding operation to ensure
Strategy operation.
It is described above, be only presently preferred embodiments of the present invention, any formal limitation not done to the present invention, it is every according to
Any simply modification, the equivalent variations made according to the technical spirit of the present invention to above example, each fall within the protection of the present invention
Within the scope of.