CN102508834A - Message-based event application device and message-based event application method - Google Patents
Message-based event application device and message-based event application method Download PDFInfo
- Publication number
- CN102508834A CN102508834A CN2011102845259A CN201110284525A CN102508834A CN 102508834 A CN102508834 A CN 102508834A CN 2011102845259 A CN2011102845259 A CN 2011102845259A CN 201110284525 A CN201110284525 A CN 201110284525A CN 102508834 A CN102508834 A CN 102508834A
- Authority
- CN
- China
- Prior art keywords
- message
- data source
- affairs
- scale
- action statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a message-based event application device, which is used for enabling incremental data manipulation statements applied to a first data source to be synchronously applied to a second data source, and comprises a dividing unit and an application unit, wherein the dividing unit is used for obtaining the incremental data manipulation statements from the first data source and packeting the incremental data manipulation statements according to transactions, judging whether the transactions are events, dividing the events into multiple parts and sending a message formed by the incremental data manipulation statements in each part into the application unit; and the application unit is used for creating local transactions of the second data source and corresponding application tasks when receiving the message, so that the application tasks apply the incremental data manipulation statements in the message to the second data source based on the local transactions. The invention further provides a message-based event application method. According to the technical scheme, synchronism of real-time incremental data can be improved.
Description
Technical field
The present invention relates to the data synchronization technology field, in particular to message based large-scale transactional applications apparatus and method.
Background technology
In data sync synoptic diagram shown in Figure 1; At first; Incremental data extraction module (SEM) 102 extracts increment operation from the OLTP data source, will do basic analysis to these data simultaneously and by the affairs classification, packs then and is transferred to data conversion module (SPM) 104; Data conversion module (SPM) 104 carries out conversion process to the SQL operation of affairs after receiving the good affairs of packing as requested, and the back notification data application module (SAM) 106 of finishing dealing with comes applied transaction.In order to guarantee synchronous absolute reliable of whole data increment, can dispose a transmission platform (for example message-oriented middleware etc.) in distributed environment lower floor, guarantee synchronous reliability through it.This is no problem in the ordinary course of things, but when to run into the affairs that are packaged into are large-scale affairs (for example a batch updating operation has 100,000,000 records), in this case, following problem can occur:
Once generate the large-scale affairs that comprise so many SQL operation, system probably directly collapses because of low memory;
Even if internal memory can be handled the message that comprises large-scale like this affairs; But by present realization mechanism all is after receiving whole affairs; SPM and SAM just can begin follow-up processing, and whole like this data sync flow process is equivalent to become serial processing, and the data net synchronization capability is also had very big influence.
Therefore, need a kind of can making and when handling large-scale affairs, can improve the data sync performance to the synchrodata treatment technology of large-scale affairs.
Summary of the invention
In view of this, technical matters to be solved by this invention is, a kind of message based large-scale transactional applications technology is provided, and utilizes this platform of message-oriented middleware to handle the remote transmission of large-scale affairs, to improve the data sync performance.
According to an aspect of the present invention; A kind of message based large-scale transactional applications device is provided, has been used for the incremental data action statement synchronous applications that is applied to first data source comprising: resolving cell in second data source; From said first data source, obtaining the incremental data action statement; Said incremental data action statement is packed by affairs, when judging said affairs and be large-scale affairs, said large-scale affairs are decomposed into many parts; Every part of said incremental data action statement that comprises is configured to a message, said message is sent to applying unit; Said applying unit; When receiving said message; Create the local matter of said second data source and create the application corresponding task, said application task is applied to said second data source based on said local matter with the incremental data action statement that comprises in the said message.
In technique scheme, preferably, said resolving cell can also comprise: the message constructing module, the said large-scale affairs after the many parts of decomposition are constituted a plurality of said message, and a plurality of said message constitute a message groups; Module is set; The attribute of each said message is set; The identity identification information of message groups, the message number in the affiliated message groups, the sequence number of said message under said attribute comprises, the identity identification information of said message groups comprise the title of said first data source and the local matter sign of said second data source.
In technique scheme, preferably, said applying unit can also comprise: second judge module; The incremental data action statement that in current said message, comprises is used when accomplishing; Judge whether said message belongs to message groups, belong to message groups if judge, then the attribute according to current said message continues to judge whether the message in the said message groups all finishes dealing with; When judged result when all finishing dealing with; Submit the local matter of said second data source to, when judged result when all not finishing dealing with, continue to handle the untreated message of residue.
In technique scheme; Preferably; Said resolving cell can also comprise: when first judge module, the number of the incremental data action statement in said affairs surpassed the threshold value of the incremental data action statement that preset said message can comprise, judging said affairs was large-scale affairs.
In technique scheme, preferably, said applying unit is created the local matter of said second data source based on the XA interface specification, and said first data source is the OLTP data source, and said second data source is the OLAP data source.
Technique scheme is through splitting the ultra-large type affairs of excavating from the OLTP data source; Can obviously improve the situation that incremental data is used: through split can control messages size, the situation that can not send data even collapse because of low memory of avoiding that system possibly occur; Setting through systematic parameter (the maximum operand that message can comprise) can adapt to hardware environment intelligently; Change original serial application into parallel the application through splitting the super large affairs, obviously improve the performance of system.
According to another aspect of the invention; A kind of message based large-scale transactional applications method also is provided; The incremental data action statement synchronous applications that is used for being applied to first data source comprises in second data source: step 402, from said first data source, obtain the incremental data action statement; Said incremental data action statement is packed by affairs; When judging said affairs and be large-scale affairs, said large-scale affairs are decomposed into many parts, every part of said incremental data action statement that comprises is configured to a message; Step 404; When receiving said message; Create the local matter of said second data source and create the application corresponding task, said application task is applied to said second data source based on said local matter with the incremental data action statement that comprises in the said message.
In technique scheme; Preferably; Said step 402 can also comprise: the said large-scale affairs after the many parts of decomposition are constituted a plurality of said message, and a plurality of said message constitute a message groups, and the attribute of each said message is set; The identity identification information of message groups, the message number in the affiliated message groups, the sequence number of said message under said attribute comprises, the identity identification information of said message groups comprise the title of said first data source and the local matter sign of said second data source.
In technique scheme; Preferably, said step 404 can also comprise: the incremental data action statement that in current said message, comprises is used when accomplishing, and judges whether said message belongs to message groups; Belong to message groups if judge; Then continue to judge whether the message in the said message groups all finishes dealing with according to the attribute of current said message, when judged result when all finishing dealing with, submit the local matter of said second data source to; When judged result when all not finishing dealing with, continue to handle the untreated message of residue.
In technique scheme; Preferably; Said step 402 can also comprise: when the number of the incremental data action statement in the said affairs surpassed the threshold value of the incremental data action statement that preset said message can comprise, judging said affairs was large-scale affairs.
In technique scheme, preferably, the local matter of said second data source is created based on the XA interface specification and is formed, and said first data source is the OLTP data source, and said second data source is the OLAP data source.
Technique scheme is through splitting the large-scale affairs of excavating from the OLTP data source; Can obviously improve the situation that incremental data is used: through split can control messages size, the situation that can not send data even collapse because of low memory of avoiding that system possibly occur; Setting through systematic parameter (the maximum operand that message can comprise) can adapt to hardware environment intelligently; Change original serial application into parallel the application through splitting the super large affairs, obviously improve the performance of system.
Description of drawings
Fig. 1 shows the data sync synoptic diagram of correlation technique;
Fig. 2 shows the block diagram of message based according to an embodiment of the invention large-scale transactional applications system;
Fig. 3 shows the block diagram of message based according to an embodiment of the invention large-scale transactional applications device;
Fig. 4 shows the process flow diagram of message based according to an embodiment of the invention large-scale transactional applications method;
Fig. 5 shows the process flow diagram that extracts the incremental data action statement according to an embodiment of the invention;
Fig. 6 shows the process flow diagram of the affairs of sending single packing according to an embodiment of the invention; And
Fig. 7 shows the process flow diagram of Application of Large affairs according to an embodiment of the invention.
Embodiment
In order more to be expressly understood above-mentioned purpose of the present invention, feature and advantage, the present invention is further described in detail below in conjunction with accompanying drawing and embodiment.
Set forth a lot of details in the following description so that make much of the present invention, still, the present invention can also adopt other to be different from other modes described here and implement, and therefore, the present invention is not limited to the restriction of following disclosed specific embodiment.
The implication of english abbreviation and being explained as follows of relevant noun that hereinafter occur:
DI:Data Integration, the abbreviation of data integration.
OLTP data source: be used for handling the professional data base management system (DBMS) of trade type in the enterprise-oriented system.
OLAP data source: be used for handling the professional data base management system (DBMS) of inquiry/analytic type in the enterprise-oriented system.Consideration for performance; Large enterprise is when the distributed service operation system of framework oneself; Generally all can be with these two types of professional separate processes, correspondingly data source also can separately be used, and strides the data sync of data source and then can the perhaps manual timing of tool using accomplish.
Affairs: be meant the set of series of operation.Affairs have atomicity, consistance, four characteristics of isolation and persistence (ACID).
SQL operation: mainly refer to professional going up here and use maximum DML operations; As insert operation (INSERT), deletion action (DELETE) and upgrade operation (UPDATE); In fact this is the form of expression of incremental data in the OLTP data source, can be updated to the legacy data in the OLAP data source up-to-date state through these operations.
The master data action statement extraction module of SQL Redo Info Extractor Module:OLTP data source is called for short SEM.
SQL Redo Info Processing Module: the processing module to the basic statement that excavates is handled, be called for short SPM.
SQL Statement Applying Module: the application module with the SQL statement of handling well is used, be called for short SAM.
Message-oriented middleware: be the basic software of a kind of framework in system bottom (between application software and operating system).
Be concrete introduction below about message-oriented middleware:
Message-oriented middleware (Message Oriented Middleware is called for short MOM), its major function is the complicacy of shielding bottom-layer network data transmission, realizes high performance reliable transmission; The data consistency when function through transaction management guarantees many message operation cooperations; For distributed environment, MOM can also carry out the linear expansion of performance flexibly and form cluster.These advantages also are to select the reason of MOM as the bottom transmission platform in the present invention.Following topmost object is arranged among the MOM:
Message: an encapsulation of data is made up of attribute head (HEAD) and data volume (BODY).Can analogize to the record in the database table.
Formation: the container of message is equivalent to the table of lane database.
The producer/message person: be responsible for sending and receiving the application of message in the point-to-point application model in the MOM system respectively.
Below in conjunction with Fig. 2 message based large-scale transactional applications according to an embodiment of the invention system is described.
In the infosystem layer, comprise OLTP data source (first data source) 110, OLAP data source (second data source) 112 and message-oriented middleware (MOM) 108, select MOM as the bottom transmission platform.System also comprises incremental data extraction module 102, data conversion module 104 and data application module 106.Incremental data extraction module 102 is responsible in the OLTP data source, extracting the incremental data operation; Become affairs through carrying out classifying packing after the fundamental analysis then; Incremental data extraction module 102 checks also whether the affairs that are packaged into belong to large-scale affairs, if large-scale affairs also split into many parts with large-scale affairs; SQL operative configuration in said every part is become a message; These message are sent to data conversion module 104, and data conversion module 104 is responsible for the operation in the message that receives is filtered and changed, and the message after will changing then sends data application module 106 to; Data application module 106 is created the local matter of OLAP, and creates corresponding task and come the operation that comprises in the application message.Therefore, in the present embodiment, the data extraction module among Fig. 1 102 is improved, enabled that the affairs of super large are carried out intelligence and split; Also data application module 106 has been carried out improving and having increased other middleware, for example message-oriented middleware enables to improve the performance of remote transmission and handles the affairs of striding a plurality of message.
In front MOM is done and described its high efficiency of transmission and the reliable characteristic of data when introducing, therefore in the present embodiment, will select MOM as the bottom transmission platform.Select MOM to also have the another one reason: after the SQL operation of super large affairs is split as transmission platform; Each part all will be packaged into a message and handle (because after message has only real enterings MOM, could real trigger network transmission perhaps the treatment scheme of message recipient) by MOM affairs independently.How the message at the SQL operation place in these transmission can become a problem associated; Here the group message mechanism that will use MOM carries out association; The group message of MOM can be striden affairs; Each message groups can have a group id (group id), and each MOM message can be interrelated by them through group id is set, with reception and the processing that makes things convenient for message.
In the incremental data extraction module, the fractionation rule of formulating large-scale affairs through the parameter of system definition is:
When data extract, in order to split large-scale affairs by expection, what need in system, define in advance is large-scale affairs.So-called large-scale affairs are meant that the SQL operation amount that is comprised has surpassed the affairs that maximum SQL that the single MOM message of system definition can comprise operates number.Can find out if defined the maximum SQL operation number that single MOM message can comprise from definition also just to have defined the super large affairs, so the parameter that in system, defines has:
The maximum SQL operation number of MOM message: this rule has been arranged; Incremental data extraction module 102 just can go to check that the affairs that will send are the super large affairs according to this parameter when the affairs packing of excavating is sent; If; Then the SQL according to this parameter-definition operates maximum number; Affairs are cut apart, then each message is submitted among the MOM and goes as a MOM affairs (like this could be in whole affairs also in when transmission, the message of coming in earlier possibly be transferred to data application module 106 and begin to have used).Better related for the message in organizing to make things convenient for SAM to handle, following attribute can be set at the head of MOM message:
Group ID: use the data source Transaction Identifier of DSN+OLTP of OLTP to come unique one group of message of confirming;
Message number in the group; And the sequence number of this message.
In data application module 106; Need to do handled: during service data in receiving message and processing messages to many message transactions; Need at first inspection whether to organize message (checking that promptly the affairs from the OLTP data source are large-scale affairs),, then after this Message Processing finishes, can not submit at once if what find to handle is large-scale affairs; Need be according to the message number in group ID and the group and the sequence number of current message; The next message that reaches in the group is then handled, and after all message in this group are all finished dealing with, submits the local matter on the OLAP data source again to.
Fig. 3 shows the block diagram of message based according to an embodiment of the invention large-scale transactional applications device.
As shown in Figure 3; Message based according to an embodiment of the invention large-scale transactional applications device 300 is used for the incremental data action statement synchronous applications that is applied to first data source comprising: resolving cell 302 (the for example data extraction module among Fig. 2 102) in second data source; From said first data source, obtaining the incremental data action statement; Said incremental data action statement is packed by affairs, when judging the said affairs that are packaged into and be large-scale affairs, said large-scale affairs are decomposed into many parts; Every part of said incremental data action statement that comprises is configured to a message, said message is sent to applying unit; Said applying unit 304 (the for example data application module 106 among Fig. 2); When receiving said message; Create the local matter of said second data source and create the application corresponding task, said application task is applied to said second data source based on said local matter with the incremental data action statement that comprises in the said message.
In technique scheme, preferably, said resolving cell 302 can also comprise: message constructing module 3022, the said large-scale affairs after the many parts of decomposition are constituted a plurality of said message, and a plurality of said message constitute a message groups; Module 3024 is set; The attribute of each said message is set; The identity identification information of message groups, the message number in the affiliated message groups, the sequence number of said message under said attribute comprises, the identity identification information of said message groups comprise the title of said first data source and the local matter sign of said second data source.
In technique scheme, preferably, said applying unit 304 can also comprise: second judge module 3042; The incremental data action statement that in current said message, comprises is used when accomplishing; Judge whether said message belongs to message groups, belong to message groups if judge, then the attribute according to current said message continues to judge whether the message in the said message groups all finishes dealing with; When judged result when all finishing dealing with; Submit the local matter of said second data source to, when judged result when all not finishing dealing with, continue to handle the untreated message of residue.
In technique scheme; Preferably; Said resolving cell 302 can also comprise: first judge module 3026; When the number of the incremental data action statement in said affairs surpassed the threshold value of the incremental data action statement that preset said message can comprise, judging said affairs was large-scale affairs.
In technique scheme, preferably, said applying unit is created the local matter of said second data source based on the XA interface specification, and said first data source is the OLTP data source, and said second data source is the OLAP data source.
Technique scheme is through splitting the large-scale affairs of excavating from the OLTP data source; Can obviously improve the situation that incremental data is used: through split can control messages size, the situation that can not send data even collapse because of low memory of avoiding that system possibly occur; Setting through systematic parameter (the maximum operand that message can comprise) can adapt to hardware environment intelligently; Change original serial application into parallel the application through splitting the super large affairs, obviously improve the performance of system.
Fig. 4 shows the process flow diagram of message based according to an embodiment of the invention large-scale transactional applications method.
As shown in Figure 4; Message based according to an embodiment of the invention large-scale transactional applications method; The incremental data action statement synchronous applications that is used for being applied to first data source comprises in second data source: step 402, from said first data source, obtain the incremental data action statement; Said incremental data action statement is packed by affairs; When judging the said affairs that are packaged into and be large-scale affairs, said large-scale affairs are decomposed into many parts, every part of said incremental data action statement that comprises is configured to a message; Step 404; When receiving said message; Create the local matter of said second data source and create the application corresponding task, said application task is applied to said second data source based on said local matter with the incremental data action statement that comprises in the said message.
In technique scheme; Preferably; Said step 402 can also comprise: the said large-scale affairs after the many parts of decomposition are constituted a plurality of said message, and a plurality of said message constitute a message groups, and the attribute of each said message is set; The identity identification information of message groups, the message number in the affiliated message groups, the sequence number of said message under said attribute comprises, the identity identification information of said message groups comprise the title of said first data source and the local matter sign of said second data source.
In technique scheme; Preferably, said step 404 can also comprise: the incremental data action statement that in current said message, comprises is used when accomplishing, and judges whether said message belongs to message groups; Belong to message groups if judge; Then continue to judge whether the message in the said message groups all finishes dealing with according to the attribute of current said message, when judged result when all finishing dealing with, submit the local matter of said second data source to; When judged result when all not finishing dealing with, continue to handle the untreated message of residue.
In technique scheme; Preferably; Said step 402 can also comprise: when the number of the incremental data action statement in the said affairs surpassed the threshold value of the incremental data action statement that preset said message can comprise, judging said affairs was large-scale affairs.
In technique scheme, preferably, the local matter of said first data source is created based on the XA interface specification and is formed, and said first data source is the OLTP data source, and said second data source is the OLAP data source.
Technique scheme is through splitting the affairs of excavating from the OLTP data source; Can obviously improve the incremental data applicable cases: through split can control messages size, the situation that can not send data even collapse because of low memory of avoiding that system possibly occur; Setting through systematic parameter (the maximum operand that message can comprise) can adapt to hardware environment intelligently; Change original serial application into parallel the application through splitting the super large affairs, obviously improve the performance of system.
Fig. 5 shows the process flow diagram that extracts the incremental data action statement according to an embodiment of the invention.
As shown in Figure 5, step 502 is at first carried out the initialization of related resource, like the connection of MOM, and the connection of data source etc.Initialization will be carried out a circular treatment flow process after accomplishing.
Step 506 if system state is normal, then gets into the OLTP incremental data and excavates flow process.Extract the SQL primitive operation of increment from the OLTP data source.
Step 508 is carried out basic filtration (one-level filtration).
Step 510 is packaged into affairs (in the daily record of OLTP, having put down in writing the information such as affiliated affairs of SQL operation) according to the classification of affairs to the increment SQL operation of excavating.
Step 512 sends to data conversion module one by one with the affairs of packing and does corresponding conversion process (affairs of accomplishing are sent in circulation).
Step 514 discharges the resource that this excavates locking, carries out the data processing of next round then.
In above-mentioned steps 512; " affairs of accomplishing are sent in circulation " is movable, and (uncompleted affairs can be buffered in this locality; After excavating all operations of affairs, just can trigger transmission flow) the flow process of concrete process of transmitting as shown in Figure 6, Fig. 6 shows the process flow diagram of the affairs of sending single packing according to an embodiment of the invention.
As shown in Figure 6, step 602, whether all can do before transmission for the affairs of each packing is the inspection of complete transaction.
Step 614 if be complete affairs, then continues to judge whether to be large-scale affairs.
Step 608 is taken out a SQL operation in the set that splits.
Step 610 becomes a message with a SQL operative configuration of taking out.Each part SQL operation for this all need be constructed a message (being a message in the group message).During for each part SQL operative configuration message of cutting apart, the attribute of group message to be set all: the message number in group id, the group and the sequence number of current message, to make things convenient for subsequent treatment.
Step 612 sends message with other modules that make things convenient for system then and handles.
Step 614 checks whether all the SQL operations in these affairs all send, if after all SQL operations in the ultra-large type affairs all sent, the epicycle processing finished.If no, then get back to step 608.
Fig. 7 shows the process flow diagram of Application of Large affairs according to an embodiment of the invention.
As shown in Figure 7, system state is monitored and obtained to step 702 after the related resource initialization is accomplished.
Step 704 judges that whether system state is for stopping, if judged result for being, then gets into step 722, if judged result then gets into step 708 for not.
Step 722 in case find that system requirements stops, then discharging relevant resource and end immediately.
Step 708 is waited for the message that comprises the SQL operation with blocking.
Step 710 is after receiving a message that comprises SQL operation, with XA (based on the XA standard) local matter of creating an OLAP.
Step 712 is created corresponding task then and is gone to use the SQL operation (these SQL operations possibly be the parts of OLTP ultra-large type affairs) that is comprised in this message.
Step 714 after the SQL operational applications that this message comprised is accomplished, will check whether this message is a group message.
Step 718 if not group message, then will be submitted the XA local matter of OLAP data source to, and this wheel operation is accomplished.
Step 718 if all message in the discovery group all are processed completion, will be submitted to or the XA local matter of rollback OLAP according to application result, and this wheel operation is accomplished.
Step 720 if also have message not to be processed in the discovery group, then need go to receive other message in untreated group according to the attribute of group message, repeats application in front and inspection logic after receiving again, and all message in group all are processed completion.
More than be described with reference to the accompanying drawings according to technical scheme of the present invention; Through the affairs of excavating from the OLTP data source are split; Can obviously improve the situation that incremental data is used: through split can control messages size, the situation that can not send even collapse because of low memory of avoiding that system possibly occur; Through the setting of systematic parameter, can adapt to hardware environment intelligently.Change original serial application into parallel the application through splitting large-scale affairs, obviously improved the performance of system.
The above is merely the preferred embodiments of the present invention, is not limited to the present invention, and for a person skilled in the art, the present invention can have various changes and variation.All within spirit of the present invention and principle, any modification of being done, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. message based large-scale transactional applications device is used for the incremental data action statement synchronous applications that is applied to first data source it is characterized in that in second data source, comprising:
Resolving cell; From said first data source, obtaining the incremental data action statement; Said incremental data action statement is packed by affairs, when judging said affairs and be large-scale affairs, said large-scale affairs are decomposed into many parts; Every part of said incremental data action statement that comprises is configured to a message, said message is sent to applying unit;
Said applying unit; When receiving said message; Create the local matter of said second data source and create the application corresponding task, said application task is applied to said second data source based on said local matter with the incremental data action statement that comprises in the said message.
2. message based large-scale transactional applications device according to claim 1 is characterized in that said resolving cell also comprises:
The message constructing module constitutes a plurality of said message with the said large-scale affairs after the many parts of decomposition, and a plurality of said message constitute a message groups;
Module is set; The attribute of each said message is set; The identity identification information of message groups, the message number in the affiliated message groups, the sequence number of said message under said attribute comprises, the identity identification information of said message groups comprise the title of said first data source and the local matter sign of said second data source.
3. message based large-scale transactional applications device according to claim 1 is characterized in that said applying unit also comprises:
Second judge module, the incremental data action statement that in current said message, comprises is used when accomplishing, and judges whether said message belongs to message groups; Belong to message groups if judge; Then continue to judge whether the message in the said message groups all finishes dealing with according to the attribute of current said message, when judged result when all finishing dealing with, submit the local matter of said second data source to; When judged result when all not finishing dealing with, continue to handle the untreated message of residue.
4. message based large-scale transactional applications device according to claim 1 is characterized in that said resolving cell also comprises:
When first judge module, the number of the incremental data action statement in said affairs surpassed the threshold value of the incremental data action statement that preset said message can comprise, judging said affairs was large-scale affairs.
5. according to each described message based large-scale transactional applications device in the claim 1 to 4; It is characterized in that; Said applying unit is created the local matter of said second data source based on the XA interface specification; Said first data source is the OLTP data source, and said second data source is the OLAP data source.
6. message based large-scale transactional applications method is used for the incremental data action statement synchronous applications that is applied to first data source it is characterized in that in second data source, comprising:
Step 202; From said first data source, obtain the incremental data action statement; Said incremental data action statement is packed by affairs; When judging said affairs and be large-scale affairs, said large-scale affairs are decomposed into many parts, every part of said incremental data action statement that comprises is configured to a message;
Step 204; When receiving said message; Create the local matter of said second data source and create the application corresponding task, said application task is applied to said second data source based on said local matter with the incremental data action statement that comprises in the said message.
7. message based large-scale transactional applications method according to claim 6; It is characterized in that; Said step 202 also comprises: the said large-scale affairs after the many parts of decomposition are constituted a plurality of said message; A plurality of said message constitute a message groups; The attribute of each said message is set, the identity identification information of message groups, the message number in the affiliated message groups, the sequence number of said message under said attribute comprises, the identity identification information of said message groups comprises the title of said first data source and the local matter sign of said second data source.
8. message based large-scale transactional applications method according to claim 7; It is characterized in that said step 204 also comprises: the incremental data action statement that in current said message, comprises is used when accomplishing, and judges whether said message belongs to message groups; Belong to message groups if judge; Then continue to judge whether the message in the said message groups all finishes dealing with according to the attribute of current said message, when judged result when all finishing dealing with, submit the local matter of said second data source to; When judged result when all not finishing dealing with, continue to handle the untreated message of residue.
9. message based large-scale transactional applications method according to claim 6; It is characterized in that; Said step 202 also comprises: when the number of the incremental data action statement in the said affairs surpassed the threshold value of the incremental data action statement that preset said message can comprise, judging said affairs was large-scale affairs.
10. according to each described message based large-scale transactional applications method in the claim 6 to 9; It is characterized in that; The local matter of said second data source is created based on the XA interface specification and is formed, and said first data source is the OLTP data source, and said second data source is the OLAP data source.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110284525.9A CN102508834B (en) | 2011-09-22 | 2011-09-22 | Message-based event application device and message-based event application method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110284525.9A CN102508834B (en) | 2011-09-22 | 2011-09-22 | Message-based event application device and message-based event application method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102508834A true CN102508834A (en) | 2012-06-20 |
| CN102508834B CN102508834B (en) | 2014-05-07 |
Family
ID=46220924
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110284525.9A Expired - Fee Related CN102508834B (en) | 2011-09-22 | 2011-09-22 | Message-based event application device and message-based event application method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102508834B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103354560A (en) * | 2013-06-28 | 2013-10-16 | 广州菁英信息技术有限公司 | Method and system for data decomposition and synchronization |
| CN104184602A (en) * | 2013-05-23 | 2014-12-03 | 中兴通讯股份有限公司 | Incremental data synchronization method, network element management system and network management system |
| CN105354087A (en) * | 2015-12-14 | 2016-02-24 | 北京交控科技有限公司 | Real-time virtual task processing method and device |
| CN106897123A (en) * | 2015-12-21 | 2017-06-27 | 阿里巴巴集团控股有限公司 | Database operation method and device |
| CN109388481A (en) * | 2018-09-21 | 2019-02-26 | 网易（杭州）网络有限公司 | A kind of transmission method of transaction information, device, calculates equipment and medium at system |
| JP2020191067A (en) * | 2019-05-21 | 2020-11-26 | ベイジン バイドゥ ネットコム サイエンス アンド テクノロジー カンパニー リミテッド | Query processing method, query processing system, server, and computer readable medium |
| CN112039753A (en) * | 2020-08-31 | 2020-12-04 | 北京百度网讯科技有限公司 | Message packing and receiving method, device, electronic equipment and medium |
| CN112883045A (en) * | 2021-03-31 | 2021-06-01 | 中国工商银行股份有限公司 | Database transaction splitting execution method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050193041A1 (en) * | 2004-02-27 | 2005-09-01 | Serge Bourbonnais | Parallel apply processing in data replication with preservation of transaction integrity and source ordering of dependent updates |
| CN101784988A (en) * | 2007-08-30 | 2010-07-21 | 国际商业机器公司 | Transaction aggregation to increase transaction processing throughput |
| CN102024052A (en) * | 2010-12-17 | 2011-04-20 | 北京握奇数据系统有限公司 | Method and device for realizing transaction of data storage system |
-
2011
- 2011-09-22 CN CN201110284525.9A patent/CN102508834B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050193041A1 (en) * | 2004-02-27 | 2005-09-01 | Serge Bourbonnais | Parallel apply processing in data replication with preservation of transaction integrity and source ordering of dependent updates |
| CN101784988A (en) * | 2007-08-30 | 2010-07-21 | 国际商业机器公司 | Transaction aggregation to increase transaction processing throughput |
| CN102024052A (en) * | 2010-12-17 | 2011-04-20 | 北京握奇数据系统有限公司 | Method and device for realizing transaction of data storage system |
Non-Patent Citations (1)
| Title |
|---|
| 闻敬谦等: "JMS在解决企业信息系统长事务中的应用", 《计算机与现代化》 * |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104184602A (en) * | 2013-05-23 | 2014-12-03 | 中兴通讯股份有限公司 | Incremental data synchronization method, network element management system and network management system |
| CN104184602B (en) * | 2013-05-23 | 2019-01-04 | 中兴通讯股份有限公司 | Incremental data synchronization method, Element management system and Network Management System |
| US10021182B2 (en) | 2013-06-28 | 2018-07-10 | Guangzhou Ucweb Computer Technology Co., Ltd. | Method and apparatus for data synchronization |
| WO2014206034A1 (en) * | 2013-06-28 | 2014-12-31 | 广州市动景计算机科技有限公司 | Data decomposition and synchronization method and system |
| CN103354560B (en) * | 2013-06-28 | 2017-05-17 | 广州市动景计算机科技有限公司 | Method and system for data decomposition and synchronization |
| CN103354560A (en) * | 2013-06-28 | 2013-10-16 | 广州菁英信息技术有限公司 | Method and system for data decomposition and synchronization |
| CN105354087B (en) * | 2015-12-14 | 2018-10-23 | 交控科技股份有限公司 | The virtual processing method and processing device of real-time task |
| CN105354087A (en) * | 2015-12-14 | 2016-02-24 | 北京交控科技有限公司 | Real-time virtual task processing method and device |
| WO2017107810A1 (en) * | 2015-12-21 | 2017-06-29 | 阿里巴巴集团控股有限公司 | Database operating method and device |
| CN106897123A (en) * | 2015-12-21 | 2017-06-27 | 阿里巴巴集团控股有限公司 | Database operation method and device |
| CN106897123B (en) * | 2015-12-21 | 2021-07-16 | 阿里巴巴集团控股有限公司 | Database operation method and device |
| CN109388481A (en) * | 2018-09-21 | 2019-02-26 | 网易（杭州）网络有限公司 | A kind of transmission method of transaction information, device, calculates equipment and medium at system |
| CN109388481B (en) * | 2018-09-21 | 2021-08-17 | 网易（杭州）网络有限公司 | Transaction information transmission method, system, device, computing equipment and medium |
| JP2020191067A (en) * | 2019-05-21 | 2020-11-26 | ベイジン バイドゥ ネットコム サイエンス アンド テクノロジー カンパニー リミテッド | Query processing method, query processing system, server, and computer readable medium |
| JP6998976B2 (en) | 2019-05-21 | 2022-01-18 | ベイジン バイドゥ ネットコム サイエンス テクノロジー カンパニー リミテッド | Query processing methods, query processing systems, servers and computer-readable media |
| CN112039753A (en) * | 2020-08-31 | 2020-12-04 | 北京百度网讯科技有限公司 | Message packing and receiving method, device, electronic equipment and medium |
| CN112883045A (en) * | 2021-03-31 | 2021-06-01 | 中国工商银行股份有限公司 | Database transaction splitting execution method and device |
| CN112883045B (en) * | 2021-03-31 | 2024-05-17 | 中国工商银行股份有限公司 | Database transaction splitting execution method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102508834B (en) | 2014-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102508834B (en) | Message-based event application device and message-based event application method | |
| CN102306200B (en) | Device and method for concurrently applying incremental data manipulation statements | |
| CN103761318B (en) | A kind of method and system of relationship type synchronization of data in heterogeneous database | |
| EP3120261B1 (en) | Dependency-aware transaction batching for data replication | |
| AU648263B2 (en) | Portable and dynamic distributed applications architecture | |
| EP2891994A1 (en) | Method for achieving automatic synchronization of multisource heterogeneous data resources | |
| CN109614432B (en) | System and method for acquiring data blood relationship based on syntactic analysis | |
| CN105320680A (en) | Data synchronization method and device | |
| CN101807210A (en) | Database data synchronic method, system and device | |
| CN102981823B (en) | A kind of domain driven development plug-in system | |
| CN101640587B (en) | Data synchronization method and data synchronization device | |
| CN114036119A (en) | Data synchronization method based on button and database log | |
| CN102722368B (en) | Plug-in software designing method based on document tree and message pump | |
| CN110837535A (en) | Data synchronization method, device, equipment and medium | |
| CN112214453B (en) | Large-scale industrial data compression storage method, system and medium | |
| CN111177244A (en) | Data association analysis method for multiple heterogeneous databases | |
| CN111694799A (en) | Parallel log analysis method and data synchronization system based on log analysis synchronization | |
| CN104392123A (en) | CDA (Clinical Document Architecture) engine system and implementation method | |
| US20180121532A1 (en) | Data table partitioning management method and apparatus | |
| CN109491988B (en) | Data real-time association method supporting full-scale updating | |
| CN114297292A (en) | Data synchronization system based on canal platform and execution method | |
| EP1505516A1 (en) | Metadata based query | |
| CN111651758B (en) | Method for auditing result set of relational database of operation and maintenance auditing system | |
| CN113177088A (en) | Multi-scale simulation big data management system for material irradiation damage | |
| CN113505175B (en) | Method for realizing MySQL incremental data synchronization based on message queue |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C56 | Change in the name or address of the patentee |
Owner name: YONYOU NETWORK TECHNOLOGY CO., LTD. Free format text: FORMER NAME: UFIDA SOFTWARE CO., LTD. |
|
| CP03 | Change of name, title or address |
Address after: 100094 Haidian District North Road, Beijing, No. 68 Patentee after: Yonyou Network Technology Co., Ltd. Address before: 100094 Beijing city Haidian District North Road No. 68, UFIDA Software Park Patentee before: UFIDA Software Co., Ltd. |
|
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20140507 Termination date: 20200922 |