CN101944114B - Data synchronization method between memory database and physical database - Google Patents
Data synchronization method between memory database and physical database Download PDFInfo
- Publication number
- CN101944114B CN101944114B CN2010102852347A CN201010285234A CN101944114B CN 101944114 B CN101944114 B CN 101944114B CN 2010102852347 A CN2010102852347 A CN 2010102852347A CN 201010285234 A CN201010285234 A CN 201010285234A CN 101944114 B CN101944114 B CN 101944114B
- Authority
- CN
- China
- Prior art keywords
- sql
- database
- grammer
- memory database
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data synchronization method between an memory database and a physical database. The method comprises the following steps: 1) representing the operation to the source database in the mode of SQL operation, decomposing the SQL operation to an SQL syntax and the corresponding SQL parameters, registering the SQL syntax and the SQL parameters of the operation source database, storing the registered SQL syntax and the SQL parameters in an SQL management region to manage; 2) for a data change of the source database, combining the SQL syntax with the SQL parameters to generate a logical log; and 3) regenerating the logical log of the step 2) in the target database according to the registered SQL syntax and the SQL parameters. The data synchronization method of the invention utilizes the prepare mechanism once and the execute mechanism many times to greatly increase the synchronization efficiency; the size of the logical log is reduced, the transmission loss of the system and the waste of the disk space are reduced; and the synchronization application and the corresponding SQL changes can be supported, and the updating and extending of the system can be supported.
Description
Technical field
The present invention relates to data processing method, relate in particular to the method for data synchronization between a kind of memory database and physical database.
Background technology
Modern communications is more and more stronger to the computer technology dependence; In the continuous development of telecom operators; Set up the various information system; Comprise Operation Network system, comprehensive accounting system, charge system, the office automation system, customer service system, big customer's management system, performance analysis and decision system or the like, these systems have accumulated the valuable data information of magnanimity, through these data are carried out data mining, query analysis and on-line analysis; Can discern the market opportunity, formulate market strategy, further improve customer relationship.But these data informations are too huge, and the record strip number is easily in hundred million; With the China Mobile is example; Its national mobile phone user's sum surpasses 300,000,000, and the userbase of many provinces company all surpasses ten million, for huge data volume like this; Traditional physical database based on disk (RMDB) management system more and more is difficult to deal with, so memory database (MDB) has occurred at field of telecommunications.
Because market and technological fast development; Telecommunication service is in continuous expansion; Its operation and management is continued to optimize; The framework of some traditional support systems can not satisfy growing business need and customer demand gradually, introduces some new technology and solves the problem that we run in producing and be inevitable.Through adopting memory database to replace former share memory technology, make to comprise non-type thing in internal memory originally interface, form and managed all standardization.
Memory database is placed on data the database of direct control in the internal memory exactly.With respect to disk, the reading and writing data speed of internal memory will exceed several magnitude, saves the data in to compare in the internal memory from the disk visit greatly to improve the performance of application.Memory database has been abandoned the traditional approach of data in magnetic disk management simultaneously; All in internal memory, designed architecture again based on total data; And aspect metadata cache, fast algorithm, parallel work-flow, also carried out corresponding improvement; So data processing speed is more a lot of soon than the data processing speed of traditional database, generally all more than 10 times.
With respect to utilizing program development means invoke memory to handle, memory database is had its own advantage.At first, memory database is the database management language of commercialization, has greatly shortened the construction cycle; Secondly, memory database has open platform and interface, and program development is convenient more flexibly with transplanting, also is convenient to safeguard and secondary development; The 3rd, can be through using the unified sql like language data in the audit memory easily; At last, can in database, ensure safety of data and integrality.These advantages safeguard it all is favourable for quick deployment and simplification.
Always in constantly changing, this just relates to data and physical database (disk database) synchronization of data problem in the memory database to data in telecommunication business data and the charge system.Data sync in the system comprises two parts: from the memory database to the physical database synchronous (reverse sync) and synchronous (forward is synchronous) from the physical database to the memory database.
1, forward is synchronous.This part data sync adopts the mode of increment list, and the newly-increased or data updated of business system or CRM (customer relation management) will be generated in the increment list of physical database, then synchronization program these data sync in memory database.Because the data volume of increment list is very little, and data sync is real-time, can not influence accounting routine.
2, reverse sync.Because valency is criticized on the charging backstage of physical database, tired account data nearly all have been loaded in the memory database, so the corresponding tables of data of physical database will be mainly used in data backup and the peripheral data query of using to memory database.In order to guarantee that periphery uses the correctness obtain data, the change formation logic daily record of memory table, then by synchronization program real-time in physical database synchronously.
Existing data sync mode need write down a large amount of logic logs, and data synchronization efficiency is not high, therefore needs improvement badly.
Summary of the invention
The object of the present invention is to provide the method for data synchronization between a kind of high efficiency memory database and physical database.
For realizing above-mentioned purpose, the present invention provides the method for data synchronization between a kind of memory database and physical database, comprising:
Wherein, in the step 3, every related SQL grammer of logic log; When carrying out the reforming of destination data storehouse; At first search the SQL grammer after the preparation,, and then carry out the SQL grammer and reformed in the destination data storehouse if not then carry out the preparation of SQL grammer through the SQL grammer of registering in the directorial area.
On the one hand, for the data sync from the memory database to the physical database:
In the step 1; When the application process that can operate memory database in the system brings into operation; This application process is at first registered SQL grammer and SQL parameter to memory database, generates a copy simultaneously and is stored in the SQL directorial area, and this copy supplies the daily record of the execution physical database usefulness of reforming;
In the step 2, the application process in the capture systems combines the form formation logic daily record of SQL parameter to the operation of memory database with the SQL grammer;
In the step 3,, the logic log in the step 2 is reformed in physical database according to the SQL grammer and the SQL parameter of registration.
Wherein, make each application process to the corresponding SQL operational group of the operation of memory database, each SQL operational group comprises several SQL operations, and each SQL operation is made up of SQL grammer and SQL parameter group.
Wherein, also comprise:, set up the relation, application process of apply names and corresponding application process and the relation of corresponding SQL operational group for all need carry out the application process of data sync in the system.
Wherein, after the formation logic daily record, system also collects the logic log that generates in the step 2: corresponding to the related affairs buffer zone of each thread of application process, be used for storing the logic log that an office generates; Use buffer zone corresponding to related one of each application process, be used for storing the logic log that this all instance of application process is generated; Each application process stores the logic log that is generated in the message queue into more then; From message queue, obtain the logic log of application process and be placed in the system buffer, and be written to the logic log in the system buffer in the journal file of memory database.
Wherein, in the step 3, at first load and resolve the journal file of memory database,, according to SQL grammer and the SQL parameter of registration, in physical database, reform the logic log of generation then through resolving the logic log that journal file obtains in the step 2 being generated.
Wherein, in the step 3, when carrying out the reforming of physical database, search the SQL grammer after the preparation in the SQL grammer of at first in directorial area, registering,, and then carry out the SQL grammer physical database is reformed if not then carry out the preparation of SQL grammer.
On the other hand, for the data sync from the physical database to the memory database:
In the step 1,, SQL syntax table and SQL parameter list are set in the SQL directorial area, are used for the SQL grammer and the SQL parameter of registration operation physical database respectively according to the journal file of physical database;
In the step 2,, the logic log table is set, when the data of physical database change,, the data that change is write the logic log table through the trigger mechanism that physical database provides according to the journal file of physical database;
In the step 3, real time scan logic log table according to the SQL grammer and the SQL parameter of registration, is reformed the logic log that step 2 is corresponding at memory database.
Wherein, in the step 3, when carrying out the reforming of memory database:
Step 3.1, get access to logic log after, obtain the SQL grammer after the preparation according to the SQL syntax table;
If step 3.2 is obtained the SQL grammer success after the preparation, directly execution in step 3.3; If obtain the SQL grammer failure after the preparation; Then in the SQL syntax table, obtain the SQL grammer, carry out the beamhouse operation of memory database then, and in the SQL parameter list, get parms; Carry out the Wire Parameters operation of memory database then, thereby accomplish reforming of memory database;
Step 3.3, split out all SQL parameters, set gradually in the SQL grammer, carry out reforming of memory database then according to logic log.
In sum, the method for data synchronization between memory database of the present invention and physical database has utilized a prepare and repeatedly execute is machine-processed, has improved synchronous efficiency greatly; Reduce the size of logic log, reduced the waste of system transmissions loss and disk space; Support the variation of synchronous applications and corresponding SQL, supported the upgrading and the expansion of system.
Description of drawings
Below in conjunction with accompanying drawing, describe in detail through specific embodiments of the invention, will make technical scheme of the present invention and other beneficial effects obvious.
In the accompanying drawing,
Fig. 1 is the process flow diagram of the method for data synchronization between memory database of the present invention and physical database;
Fig. 2 is the structural representation of reverse sync system of having used a preferred embodiment of the method for data synchronization between memory database of the present invention and physical database;
Fig. 3 is the DFD of the reverse sync of reverse sync system shown in Figure 2;
Fig. 4 is the DFD of forward synchro system of having used a preferred embodiment of the method for data synchronization between memory database of the present invention and physical database.
Embodiment
Process flow diagram for the method for data synchronization between memory database of the present invention and physical database shown in Figure 1.This method comprises: step 1, represent the operation to source database with the form of SQL operation; The SQL operation is decomposed into SQL grammer and corresponding SQL parameter; The SQL grammer of registration operation source database and SQL parameter, and SQL grammer and the SQL parameter of registration be stored in the SQL directorial area manage; Step 2, for the data movement of source database, combine the form formation logic daily record of SQL parameter with the SQL grammer; Step 3, according to the registration SQL grammer and SQL parameter, the logic log in the step 2 is reformed in the destination data storehouse.
In the step 3; The related SQL grammer of every logic log when carrying out the reforming of destination data storehouse, is at first searched the SQL grammer after the preparation through the SQL grammer of registering in the directorial area; If not then carry out the preparation of SQL grammer, and then carry out the SQL grammer and reformed in the destination data storehouse.The present invention is through splitting into SQL statement SQL grammer and dynamic parameter, and logic log is made up of SQL grammer numbering and dynamic parameter, has reduced the size of logic log so greatly, has reduced the waste of system transmissions loss and disk space.Make full use of a prepare (preparation) of SQL grammer and execute (execution) mechanism repeatedly simultaneously, improved synchronous efficiency greatly, reduced the time delay of data sync.
Method of data synchronization between memory database of the present invention and physical database is applicable to the system that has memory database and physical database simultaneously; Such system can be made up of some main frames; Memory database and physical database can be provided with simultaneously and be also can be separately positioned on the different main frames on the main frame.For example, such system can be the business data and the charge system of telecommunications.Can support the operation of various application (application program) in the system, the application that is in running status in the system is called application process.Method of data synchronization between memory database of the present invention and physical database is supported from the memory database to the physical database synchronous (reverse sync) and synchronous (forward is synchronous) from the physical database to the memory database simultaneously.
The structural representation of reverse sync system for the preferred embodiment of having used the method for data synchronization between memory database of the present invention and physical database shown in Figure 2.This preferred embodiment is through using method of the present invention; In the system that has between memory database and physical database, realized supporting that the reverse sync and the forward of data are synchronous; According to the direction of data sync, this preferred embodiment can be divided into reverse sync system and forward synchro system two parts.
At first specify the reverse sync system of this preferred embodiment with reference to Fig. 2.
1 reverse sync
In this preferred embodiment, the daily record that reverse sync the is based on database mechanism of reforming realizes, thereby can support the hybrid programming pattern of multi-process and multithreading.Reverse sync program (reverse sync system) externally provides operation memory database API (Application Programming Interface; API) and secondary development bag (the mdbutil kit of dynamic base; Be used to connect memory database), use based on the reverse sync program (reverse sync system) of this interface exploitation and support memory database synchronous to physical database.Application process in the system (application 1) is to the operation of memory database; Caught and be recorded in the logic log of memory database by internal storage data mdbutil kit, carry out the persistence (writing journal file) of daily record through reverse sync program (reverse sync system) then and the daily record of physical database is reformed.The mdbutil kit is made up of application management, SQL management and log management three big modules.All management information all is stored in the shared memory of system, has guaranteed the safety of system information.The structure of reverse sync system specifies as follows.
1.1 system architecture
1.1.1 application management
All need synchronous application process in the management host (system), for all need carry out the application process of data sync in the system, safeguard the relation of relation, application process and the corresponding SQL operational group of setting up apply names and corresponding application process.At first call the initialization action of mdbutil during the reverse sync program start, corresponding simultaneously apply names, progress information are registered in the synchro system.
1.1.2SQL management
The SQL administration module is responsible for SQL grammer and the management of corresponding parameter in the memory database.That is to say; Corresponding to step 1 of the present invention, the SQL statement of expression SQL operation is decomposed into the form of SQL grammer and corresponding parameters, simultaneously; When the application beginning forms application process in internal memory; The SQL grammer that possibly use during this apply changes memory database of application process registration and corresponding parameters generate a copy simultaneously and are stored in the SQL directorial area to memory database, the daily record of this copy confession execution physical database usefulness of reforming.Describe with sql like language, the main data structure in the SQL directorial area is following:
//SQL grammer
typedef?struct
{
TUINT8id:8; // application numbers
TUINT8oper:8; // operation sequence number
TINT8sql [MDBUTIL_SYNC_SQL_LEN-2]; //SQL statement
}SSQLSyntax；
//SQL parameter
typedef?struct
{
TUINT32type; // parameter type
TUINT32size; // parameter size
}SSQLParam；
//SQL parameter group
typedef?struct
{
TINT32num; // number of parameters
TINT32unused; // placeholder
SSQLParam?array[MDBUTIL_SYNC_SQL_PARAM_NUM]；
}SSQLParams；
//SQL operation
typedef?struct
{
SSQLSyntax syntax; //SQL grammer
SSQLParams params; //SQL parameter
}SDMLOper；
//SQL operational group
typedef?struct
{
SSemLock lock; // atomic lock
TUINT32 id:8; / application numbers
TUINT32 flag:8; // service marking
TUINT32 num:16; //SQL number
TUINT32 unused; // placeholder
SDMLOper?group[MDBUTIL_SYNC_SQL?KIND_SIZE]；
}SDMLOperGroup；
The definition of data structure is ascending, makes each application process all can a corresponding SQL operational group (SDMLOperGroup) to the operation of memory database.Each SQL operational group comprises several SQL operations, and each SQL operation all is made up of SQL grammer (SSQLSyntax) and SQL parameter group (SSQLParams).
1.1.3 log management
Main generation, the packing (collection of logic log) of being responsible for logic log, land (generate journal file, journal file is the comprehensive of all logic logs) and reform (to physical database).Corresponding to step 2 of the present invention, the application process in the reverse sync program capture systems combines the form formation logic daily record of SQL parameter to the operation of memory database with the SQL grammer.Describe with sql like language, the organization definition of the logic log that the present invention adopted is following:
// logic log head
typedef?struct
{
TUINT32 id:8; // application numbers
TUINT32 oper:8; //SQL numbering
TUINT32 len:16; // daily record length
TUINT32 seq; // log-sequence numbers
}SLogicHead；
//type-length-value
typedef?struct
{
TUINT32 type:4; // data type
TUINT32 len:12; // data length
TUINT8 value [2]; // numerical value
TUINT8?unused[MDBUTIL_SYNC_PARAM_LEN]；
}STLV；
// logic log body
typedef?struct
{
STLV tlv [1]; // array
}SLogicBody；
// logic log
typedef?struct
{
SLogicHead?head；
SLogicBody?body；
}SLogicLog；
// public buffer head
typedef?struct
{
TUINTL id; // numbering
TUINT32 num; // number
TUINT32 len; // length
}SLogBufHead；
// affairs logic log buffer head
typedef?struct
{
SLogBufHead com; // public part
TUINT32 start; // the start time
TUINT32 group; // operational group
TUINT64?unused；
}STransLogBufHead；
// affairs logic log buffering
typedef?struct
{
STransLogBufHead head; // sew drift
TINT8buf [MDBUTIL_SYNC_PLACE HOLD]; // data buffering
}STransLogBuf；
Wherein, logic log is the least unit of operating database, and transaction journal is the least unit of synchronous operation, and transaction journal is made up of one or more logic logs.
1.1.3.1 logic log
Logic log is made up of daily record head and daily record body two parts, and wherein the daily record body is made up of one group of parameter (number can be 0), and each parameter is stored by TLV (type-length-value, type one length-value) structure.The daily record head is made up of id (numbering), oper (operation), len (length) and seq (sequence number), and wherein id and oper are used to refer to unique SQL operation, the length of len presentation logic daily record, and seq representes the sequence number in the affairs.
1.1.3.2 transaction journal
Affairs are combinations of logic log, are made up of buffer head and buffer zone two parts.Buffer head is made up of id, num (number), len, start (start time) and group (operational group), and wherein id is an affairs sequence number of carrying out memory database.Logic log is to store in the buffer zone according to execution sequence.
According to step 3 of the present invention; The formation logic daily record can be reformed in physical database according to SQL grammer of registering in the SQL directorial area and SQL parameter; Because the formation logic daily record combines the structure of SQL parameter for the SQL grammer; Make full use of a prepare (preparation) of SQL grammer and execute (execution) mechanism repeatedly, improved synchronous efficiency greatly.
It more than is the system architecture of reverse sync system; Explained among the present invention and used and SQL statement (being decomposed into grammer and parameter); The relation of SQL statement and logic log; Application is represented with the SQL statement that is decomposed into SQL grammer and parameter the operation of memory database, uses SQL grammer and parameter to generate logic log simultaneously.Compared to existing method of data synchronization, the dynamic increase that synchronization program support of the present invention is used and SQL operates, thus supported that system carries out level and smooth upgrading.Through the fractionation of SQL grammer and parameter, reduced data transmission greatly, improved synchronous execution efficient.
For process synchronous from the memory database to the physical database, it is whole synchronous core content that logic log is reformed, and logic log packing is arranged, logic log land and logic log in execution of physical database or the like.
As shown in Figure 3, it is the DFD of the reverse sync of reverse sync system shown in Figure 2, shows synchronous mode of operation through dynamic logic log among Fig. 3.
1.2 data circulation
If data stream alternation of hosts's expression daily record collection process from less to more, buffering (T Buf) from the logic log to affairs arrives and uses buffering (App Buf); Arrive message queue (Queue) again; To system buffer (SysBuf),, finally implement physical database to journal file (Log file).Whole collection process is similar to the formation of seawater, brook, river, Jiang Hehai.
At first, be used for storing the logic log of affairs corresponding to the related affairs buffer zone of each thread of application process (application 1, application 2 ...); There is one to use buffer zone, the daily record that is used for storing these all instances of application corresponding to each application; Store logic log in the message queue into again by each application then.The reverse sync program is obtained application log and is placed in the system buffer (SysBuf) from message queue, and is written to the logic log in the system buffer in the journal file.Last synchronization program loads and resolves journal file, in physical database, reforms logic log.
The whole circulation process of data has made full use of a prepare (preparation) of SQL grammer and execute (execution) mechanism repeatedly.Improve synchronous efficiency greatly, reduced the time delay of data sync.Every logic log can related SQL grammer, when carrying out the physical database operation, at first searches the SQL grammer behind the prepare, if not then carry out the prepare of SQL grammer, and then carries out SQL grammer execute.Through the delay of SQL grammer, supported the dynamic increase of application and SQL.
1.3 dynamic synchronization
With reference to existing data synchronous system, if the logic log of catching is SQL statement (not having dynamic parameter), system itself just supports synchronously dynamic so.But the existing this method of synchronization has increased record logic log amount on the one hand, has wasted a large amount of disk storage spaces; Do not make full use of prepare of database and execute mechanism repeatedly simultaneously, data synchronization efficiency is not high, is not enough to support real-time transaction system and uses.
And in the present invention, at first through splitting into SQL statement SQL grammer and dynamic parameter, logic log is made up of SQL grammer numbering and dynamic parameter.Reduce the size of logic log so greatly, reduced the waste of system transmissions loss and disk space.Prepare and execute mechanism have repeatedly been utilized simultaneously; Improved synchronous efficiency greatly; When carrying out the reforming of physical database; Search the SQL grammer after the preparation in the SQL grammer of at first in directorial area, registering,, and then carry out the SQL grammer physical database is reformed if not then carry out the preparation of SQL grammer.
Secondly the variation of synchronous applications and corresponding SQL has been supported in the dynamic increase of application and SQL among the present invention, has supported the upgrading and the expansion of system; Through postponing the prepare of physical database SQL grammer, guaranteed that initiate SQL grammer can reform.The prepare of the dynamic increase of SQL and delay SQL grammer has accomplished reverse dynamic synchronization jointly.Whole in fact synchronizing process also is many execute of a prepare.
2 forwards are synchronous
In this preferred embodiment, the trigger mechanism that forward synchro system (forward synchronization program) is based on physical database realizes that this is because the storage mode and the mode of operation of physical database are different from memory database.At first, because the performance of physical database own is not high, the performance loss of trigger itself is little to the influence of physical database application performance; Secondly; The operation entry of physical database is more; OCI interface of foreground, database terminal, application (ORACLE calling interface) or the like can be as the operation entry of physical database, so trigger can be used as unique inlet of physical database data movement.When the data of physical database change,, the data that change are write the logic log table of physical database through the trigger mechanism that physical database self provides.Then by the logic log table of forward synchronization program (forward synchro system) real time scan physical database, and aim at memory database to counterlogic day and reform.
As shown in Figure 4, it is the DFD of forward synchro system of having used a preferred embodiment of the method for data synchronization between memory database of the present invention and physical database.In this preferred embodiment; Use (OCI interface of foreground, database terminal, application or the like) is reflected to physical database through trigger mechanism to the operation of physical database logic log table; Forward synchronization program (forward synchro system) is got in touch through interface table and physical database, and the interface table is made up of the logic log table of physical database and the SQL syntax table and the SQL parameter list of SQL directorial area.
During data sync from the physical database to the memory database; According to step 1 of the present invention; According to the journal file of physical database, SQL syntax table and SQL parameter list are set in the SQL directorial area, be used for the SQL grammer and the SQL parameter of registration operation physical database respectively; According to step 2 of the present invention,, the logic log table is set then, when the data of physical database change,, the data that change is write the logic log table through the trigger mechanism that physical database provides according to the journal file of physical database; According to step 3 of the present invention, real time scan logic log table according to the SQL grammer and the SQL parameter of registration, is reformed the logic log that step 2 is corresponding at memory database.In this preferred embodiment, the design of interface table is following:
2.1 interface table design
2.1.1 logic log table
2.1.2SQL syntax table
2.1.3SQL parameter list
2.1 the data stream when carrying out reforming of memory database according to step 3 of the present invention
2.1.1 after the forward synchronization program gets access to logic log, obtain the SQL grammer behind the prepare according to the TABLE_ID in the SQL syntax table and OPR_TYPE.
2.1.2 obtain the SQL grammer success behind the prepare, directly carry out next step.Otherwise, obtain the SQL grammer according to TABLE_ID and OPR_TYPE at the SQL syntax table, carry out the prepare operation of memory database then; In the SQL parameter list, get parms, carry out the Wire Parameters operation of memory database then successively, thereby accomplish reforming of memory database.
2.1.3 according to the SYN_DATA in the logic log; Split out all parameters, set gradually in the SQL grammer, carry out the execute operation of memory database then; Finally accomplished reforming of internal storage data, made the data of memory database and the data sync in the physical database.
Method of data synchronization between memory database of the present invention and physical database has been realized the data bridge of memory database and physical database, and the unified data view externally is provided; Through between application and database, adding the Data Control layer, can when accomplishing normal data manipulation, copy to data movement in the destination data storehouse.
In sum, the method for data synchronization between memory database of the present invention and physical database has utilized a prepare and repeatedly execute is machine-processed, has improved synchronous efficiency greatly; Reduce the size of logic log, reduced the waste of system transmissions loss and disk space; Support the variation of synchronous applications and corresponding SQL, supported the upgrading and the expansion of system.
The above; For the person of ordinary skill of the art; Can make other various corresponding changes and distortion according to technical scheme of the present invention and technical conceive, and all these changes and distortion all should belong to the protection domain of accompanying Claim of the present invention.
Claims (7)
1. the method for data synchronization between memory database and physical database is characterized in that, comprising:
Step 1, represent operation to source database with the form of SQL operation; The SQL operation is decomposed into SQL grammer and corresponding SQL parameter; The SQL grammer of registration operation source database and SQL parameter, and SQL grammer and the SQL parameter of registration be stored in the SQL directorial area manage;
Step 2, for the data movement of source database, combine the form formation logic daily record of SQL parameter with the SQL grammer;
Step 3, according to the registration SQL grammer and SQL parameter, the logic log in the step 2 is reformed in the destination data storehouse;
In the step 3; The related SQL grammer of every logic log when carrying out the reforming of destination data storehouse, is at first searched the SQL grammer after the preparation through the SQL grammer of registering in the directorial area; If not then carry out the preparation of SQL grammer, and then carry out the SQL grammer and reformed in the destination data storehouse;
For the data sync from the memory database to the physical database:
In the step 1; When the application process that can operate memory database in the system brings into operation; This application process is at first registered SQL grammer and SQL parameter to memory database, generates a copy simultaneously and is stored in the SQL directorial area, and this copy supplies the daily record of the execution physical database usefulness of reforming;
In the step 2, the application process in the capture systems combines the form formation logic daily record of SQL parameter to the operation of memory database with the SQL grammer;
In the step 3,, the logic log in the step 2 is reformed in physical database according to the SQL grammer and the SQL parameter of registration;
After the formation logic daily record, system also collects the logic log that generates in the step 2: corresponding to the related affairs buffer zone of each thread of application process, be used for storing the logic log that an office generates; Use buffer zone corresponding to related one of each application process, be used for storing the logic log that this all instance of application process is generated; Each application process stores the logic log that is generated in the message queue into more then; From message queue, obtain the logic log of application process and be placed in the system buffer, and be written to the logic log in the system buffer in the journal file of memory database.
2. the method for data synchronization between memory database as claimed in claim 1 and physical database; It is characterized in that; Make each application process to the corresponding SQL operational group of the operation of memory database; Each SQL operational group comprises several SQL operations, and each SQL operation is made up of SQL grammer and SQL parameter group.
3. the method for data synchronization between memory database as claimed in claim 2 and physical database; It is characterized in that; Also comprise:, set up the relation, application process of apply names and corresponding application process and the relation of corresponding SQL operational group for all need carry out the application process of data sync in the system.
4. the method for data synchronization between memory database as claimed in claim 1 and physical database; It is characterized in that; In the step 3, at first load and resolve the journal file of memory database, through resolving the logic log that journal file obtains in the step 2 being generated; According to the SQL grammer and the SQL parameter of registration, in physical database, reform the logic log that generates then.
5. the method for data synchronization between memory database as claimed in claim 1 and physical database; It is characterized in that; In the step 3, when carrying out the reforming of physical database, search the SQL grammer after the preparation in the SQL grammer of at first in directorial area, registering; If not then carry out the preparation of SQL grammer, and then carry out the SQL grammer physical database is reformed.
6. the method for data synchronization between memory database as claimed in claim 1 and physical database is characterized in that, for the data sync from the physical database to the memory database:
In the step 1,, SQL syntax table and SQL parameter list are set in the SQL directorial area, are used for the SQL grammer and the SQL parameter of registration operation physical database respectively according to the journal file of physical database;
In the step 2,, the logic log table is set, when the data of physical database change,, the data that change is write the logic log table through the trigger mechanism that physical database provides according to the journal file of physical database;
In the step 3, real time scan logic log table according to the SQL grammer and the SQL parameter of registration, is reformed the logic log that step 2 is corresponding at memory database.
7. the method for data synchronization between memory database as claimed in claim 6 and physical database is characterized in that, in the step 3, when carrying out the reforming of memory database:
Step 3.1, get access to logic log after, obtain the SQL grammer after the preparation according to the SQL syntax table;
If step 3.2 is obtained the SQL grammer success after the preparation, directly execution in step 3.3; If obtain the SQL grammer failure after the preparation; Then in the SQL syntax table, obtain the SQL grammer, carry out the beamhouse operation of memory database then, and in the SQL parameter list, get parms; Carry out the Wire Parameters operation of memory database then, thereby accomplish reforming of memory database;
Step 3.3, split out all SQL parameters, set gradually in the SQL grammer, carry out reforming of memory database then according to logic log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102852347A CN101944114B (en) | 2010-09-16 | 2010-09-16 | Data synchronization method between memory database and physical database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102852347A CN101944114B (en) | 2010-09-16 | 2010-09-16 | Data synchronization method between memory database and physical database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101944114A CN101944114A (en) | 2011-01-12 |
| CN101944114B true CN101944114B (en) | 2012-07-25 |
Family
ID=43436105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010102852347A Active CN101944114B (en) | 2010-09-16 | 2010-09-16 | Data synchronization method between memory database and physical database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101944114B (en) |
Families Citing this family (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289469B (en) * | 2011-07-26 | 2013-01-30 | 国电南瑞科技股份有限公司 | Method for supporting universal database to synchronize data based on physical isolating equipment |
| CN103136231B (en) * | 2011-11-25 | 2016-03-02 | 中国移动通信集团江苏有限公司 | Method of data synchronization between a kind of heterogeneous database and system |
| CN102937955A (en) * | 2011-11-29 | 2013-02-20 | Ut斯达康通讯有限公司 | Main memory database achieving method based on My structured query language (SQL) double storage engines |
| CN102393860A (en) * | 2011-12-06 | 2012-03-28 | 方正国际软件有限公司 | Database data migration system |
| CN103164384B (en) * | 2011-12-15 | 2016-05-18 | 中国银联股份有限公司 | The synchronization realizing method of multi-computer system shared drive and system thereof |
| CN103870357A (en) * | 2012-12-17 | 2014-06-18 | 中国移动通信集团河南有限公司 | Method and system for carrying out data replication |
| CN103488690B (en) * | 2013-09-02 | 2017-06-30 | 用友网络科技股份有限公司 | Data integrated system and data integrating method |
| CN104408054B (en) * | 2014-10-29 | 2017-10-31 | 深圳市金证科技股份有限公司 | A kind of data base management system |
| CN106919494B (en) * | 2015-12-25 | 2020-07-17 | 阿里巴巴集团控股有限公司 | Method and device for realizing android application log |
| CN105701190A (en) * | 2016-01-07 | 2016-06-22 | 深圳市金证科技股份有限公司 | Data synchronizing method and device |
| CN108073660B (en) * | 2016-11-18 | 2021-10-15 | 腾讯科技（深圳）有限公司 | Method and system for realizing cache data synchronization |
| CN106709043A (en) * | 2016-12-30 | 2017-05-24 | 江苏瑞中数据股份有限公司 | Data synchronous loading method based on database log |
| CN106874389B (en) * | 2017-01-11 | 2023-04-07 | 腾讯科技（深圳）有限公司 | Data migration method and device |
| TWI735518B (en) * | 2017-01-24 | 2021-08-11 | 香港商阿里巴巴集團服務有限公司 | Service provision method and system of database |
| CN107679232A (en) * | 2017-10-24 | 2018-02-09 | 安徽马钢自动化信息技术有限公司 | A kind of synchronous method of SAP HANA source databases and other databases |
| CN107870826A (en) * | 2017-11-07 | 2018-04-03 | 西安科技大学 | A kind of safety management system based on database detection function |
| CN110807065A (en) * | 2019-10-30 | 2020-02-18 | 浙江大华技术股份有限公司 | Memory table implementation method, memory and data node of distributed database |
| CN111930686B (en) * | 2020-07-15 | 2024-02-20 | 杭州海康威视系统技术有限公司 | Method and device for storing logs and computer equipment |
| CN113626423B (en) * | 2021-06-29 | 2024-01-30 | 欧电云信息科技(江苏)有限公司 | Log management method, device and system of business database |
| CN113792340B (en) * | 2021-09-09 | 2023-09-05 | 烽火通信科技股份有限公司 | Method and device for auditing logical logs of database |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6405205B1 (en) * | 1999-03-03 | 2002-06-11 | Nec Corporation | Message display method and system for reproduction of DML objects in relational databases |
| CN101187888A (en) * | 2007-12-11 | 2008-05-28 | 浪潮电子信息产业股份有限公司 | Method for coping database data in heterogeneous environment |
| CN101369283A (en) * | 2008-09-25 | 2009-02-18 | 中兴通讯股份有限公司 | Data synchronization method and system for internal memory database physical data base |
-
2010
- 2010-09-16 CN CN2010102852347A patent/CN101944114B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6405205B1 (en) * | 1999-03-03 | 2002-06-11 | Nec Corporation | Message display method and system for reproduction of DML objects in relational databases |
| CN101187888A (en) * | 2007-12-11 | 2008-05-28 | 浪潮电子信息产业股份有限公司 | Method for coping database data in heterogeneous environment |
| CN101369283A (en) * | 2008-09-25 | 2009-02-18 | 中兴通讯股份有限公司 | Data synchronization method and system for internal memory database physical data base |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101944114A (en) | 2011-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101944114B (en) | Data synchronization method between memory database and physical database | |
| CN102831120B (en) | A kind of data processing method and system | |
| US9619512B2 (en) | Memory searching system and method, real-time searching system and method, and computer storage medium | |
| WO2017096940A1 (en) | Data import method for spark-sql-based big-data processing platform | |
| CN102521406B (en) | Distributed query method and system for complex task of querying massive structured data | |
| CN104933160B (en) | A kind of ETL frame design method towards safety monitoring business diagnosis | |
| CN105550293A (en) | Background refreshing method based on Spark-SQL big data processing platform | |
| CN107220310A (en) | A kind of database data management system, method and device | |
| CN101799808A (en) | Data processing method and system thereof | |
| CN113419823B (en) | Alliance chain system suitable for high concurrency transaction and design method thereof | |
| CN102779138B (en) | The hard disk access method of real time data | |
| CN104317957B (en) | A kind of open platform of report form processing, system and report processing method | |
| CN102214236A (en) | Method and system for processing mass data | |
| CN101794247A (en) | Real-time database failure recovery method under nested transaction model | |
| CN105183698A (en) | Control processing system and method based on multi-kernel DSP | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN114647716B (en) | System suitable for generalized data warehouse | |
| CN115033646B (en) | Method for constructing real-time warehouse system based on Flink and Doris | |
| CN102033930A (en) | Distributed memory database system | |
| CN101839997A (en) | Seismic data parallel storage system | |
| CN110083600A (en) | A kind of method, apparatus, calculating equipment and the storage medium of log collection processing | |
| CN106294205A (en) | caching data processing method and device | |
| CN115687468A (en) | System for processing data in distributed service by ETL process button | |
| CN107451211B (en) | A kind of download system based on RabbitMQ and MongoDB | |
| CN112748997A (en) | Workflow scheduling method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |