CN105243067A - Method and apparatus for realizing real-time increment synchronization of data - Google Patents
Method and apparatus for realizing real-time increment synchronization of data Download PDFInfo
- Publication number
- CN105243067A CN105243067A CN201410321182.2A CN201410321182A CN105243067A CN 105243067 A CN105243067 A CN 105243067A CN 201410321182 A CN201410321182 A CN 201410321182A CN 105243067 A CN105243067 A CN 105243067A
- Authority
- CN
- China
- Prior art keywords
- data
- relevant database
- module
- database
- operation log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and an apparatus for realizing real-time increment synchronization of data. The method for realizing the real-time increment synchronization of the data comprises: according to table structure information of a relational database, generating a mapping relation file corresponding to the relational database in a distributed column-oriented open source database HBase; obtaining an operation log of the relational database in real time; according to the obtained operation log, obtaining change data of the relational database, and according to the established mapping relation file, updating the obtained change data into the HBase of Hadoop. The real-time increment update synchronization of the data from the relational database to Hadoop is realized, so that the burden of a Hadoop platform is effectively reduced and the user experience is improved.
Description
Technical field
The present invention relates to large data technique field, espespecially a kind of method and device realizing real-time incremental synchrodata.
Background technology
The fast development of internet, creates the mass data that volume increases sharply, the appearance of mass data and the change of data structure, brings huge challenge to all trades and professions in management and analyzing and processing.Traditional disposal route based on relational data database data effectively cannot store the miscellaneous service data growing with analyzing and processing.For this reason, many industries start to adopt distributed system architecture (Hadoop) to carry out analyzing and processing to data.Current main flow the disposable full dose that relevant database data syn-chronization realizes data to the method for Hadoop platform mainly through Sqoop is imported.Sqoop is the efficient transfer data instrument that one is present between relevant database and distributed file system (HDFS), the data of HDFS by the data importing in a relevant database in the HDFS of Hadoop, also can be led in relevant database by it.
When the data of relevant database change, to realize in the renewal data importing Hadoop in relevant database, then timing is needed to carry out full dose importing to the data in relevant database.The implication that wherein full dose imports is the total data existed in real time in relevant database imported in Hadoop.So not only increase the weight of the burden of Hadoop distributed system, and seriously consuming time.And there also do not have a kind of method can realize data to be at present synchronous to the real-time incremental renewal of Hadoop by relevant database, the change data namely in a synchronized relation type database are in Hadoop.
Summary of the invention
For solving the problem, the invention provides a kind of method and the device that realize real-time incremental synchrodata, data can be realized synchronous to the real-time incremental renewal of Hadoop by relevant database, effectively alleviate the burden of Hadoop platform simultaneously, strengthen Consumer's Experience.
In order to reach foregoing invention object, the invention discloses a kind of method realizing real-time incremental synchrodata, being applied to the data importing of relevant database to distributed system architecture, comprising:
According to the list structure information of relevant database, in the PostgreSQL database HBase towards row, generate the mapping relations file corresponding with above-mentioned relation type database;
The Operation Log of Real-time Obtaining above-mentioned relation type database;
According to the Operation Log obtained, obtain the change data of above-mentioned relation type database, according to the above-mentioned mapping relations file set up, by the change Data Update of acquisition in the HBase of Hadoop.
Further, the identify label of pre-configured above-mentioned relation type database and initiation site; The Operation Log of above-mentioned Real-time Obtaining relevant database comprises:
According to above-mentioned identify label and initiation site, from initiation site, obtain the Operation Log of relevant database corresponding to above-mentioned identify label.
Further, the Operation Log of above-mentioned acquisition above-mentioned relation type database comprises:
Receive the change data of the Operation Log of relevant database corresponding to above-mentioned identify label, and the change data received are stored in a message queue in order; Or,
When not receiving the change data obtained in change request of data and message queue and exceeding threshold value, then the change data in above-mentioned message queue are stored in corresponding catalogue file in order.
Further, after the Operation Log of above-mentioned Real-time Obtaining relevant database, said method also comprises:
Upgrade the initiation site of above-mentioned relation type database;
According to the initiation site after renewal, obtain the next Operation Log of above-mentioned relation type database.
Further, said method also comprises: the above-mentioned change data obtained be kept in local file, and record more new historical.
The invention also discloses a kind of device realizing real-time incremental synchrodata, be applied to the data importing of relevant database to distributed system architecture, comprise: build table module, log acquisition module, multiple daily record resolution client module and data update module, wherein:
Build table module, for the list structure information according to relevant database, distributed, in the PostgreSQL database HBase of row, generate the mapping relations file corresponding with above-mentioned relation type database;
Log acquisition module, for the Operation Log of Real-time Obtaining above-mentioned relation type database;
Each daily record resolution client module, is connected with above-mentioned log acquisition module respectively, for receiving the Operation Log of above-mentioned log analyzing module transmission and changing data, and the change data of acquisition is sent to data update module;
Data update module, for the change data that the above-mentioned each daily record resolution client module of real-time reception sends, and according to the above-mentioned above-mentioned mapping relations file built table module and set up, by the change Data Update of acquisition in the HBase of Hadoop.
Further, above-mentioned log acquisition module specifically for:
The identify label that pre-configured above-mentioned relation type database is unique and initiation site;
According to above-mentioned identify label and initiation site, from initiation site, obtain the Operation Log of the corresponding relevant database of above-mentioned identify label.
Further, above-mentioned log acquisition module also for:
Receive the data of the Operation Log of relevant database corresponding to above-mentioned identify label, and the data received are stored in order in a message queue, for corresponding above-mentioned daily record resolution client module acquisition request; Or,
If the data in above-mentioned message queue when the data in message queue exceed threshold value, are then stored in corresponding catalogue file by the non-acquisition request of above-mentioned daily record resolution client module above-mentioned data in order.
Further, above-mentioned log acquisition module also for:
Upgrade the initiation site of above-mentioned relation type database;
According to the initiation site after renewal, obtain the next Operation Log of above-mentioned relation type database.
Further, above-mentioned daily record resolution client module also for: when above-mentioned data update module does not start, the above-mentioned change data received are stored in local file.
Further, the above-mentioned change data obtained are kept in local file, and record more new historical.
The method of real-time incremental synchrodata that what technical scheme provided realize, be applied to the data importing of relevant database to distributed system architecture, comprise: according to the list structure information of relevant database, distributed, in the PostgreSQL database HBase of row, generate the mapping relations file corresponding with described relevant database; The Operation Log of relevant database described in Real-time Obtaining; According to the Operation Log obtained, obtain the change data of described relevant database, according to the described mapping relations file set up, by the change Data Update of acquisition in the HBase of Hadoop.The technical scheme of the application achieves data and upgrades synchronous by relevant database to the real-time incremental of Hadoop, effectively alleviates the burden of Hadoop platform simultaneously, strengthens Consumer's Experience.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide a further understanding of the present invention, and form a application's part, schematic description and description of the present invention, for explaining the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings:
Fig. 1 is the method flow diagram that the present invention realizes real-time incremental synchrodata;
Fig. 2 is the composition structural representation that the present invention realizes the device of real-time incremental synchrodata;
Embodiment
Below in conjunction with drawings and the specific embodiments, the present invention is described in detail.
Fig. 1 is the method flow diagram that the present invention realizes real-time incremental synchrodata, is applied to the data importing of relevant database to distributed system architecture (Hadoop), as shown in Figure 1, comprises the following steps:
Step 101, according to the list structure information of relevant database, distributed, in the PostgreSQL database (HBase) of row, generate the mapping relations file corresponding with this relevant database.
It should be noted that, in this step, the contingency table corresponding with relevant database can also be generated in Tool for Data Warehouse (Hive) with HBase.This contingency table refers to the tables of data consistent with list structure in relevant database created in Hive with HBase.After historical data imports to Hadoop platform, in relevant database, the renewal of tables of data can be updated in the correspondence table in Hive and HBase.HBase is used for data and stores, and Hive provides query function.This contingency table creates by Hive script.
Well-known to those skilled in the art about the method how generating mapping relations file in this step, namely by program from relevant database and available, in other words conj.or perhaps after being connected to relevant database, performing some SQL statement by corresponding interface can obtain.Be specially: after obtaining the list structure information of relevant database, the respective column name (as can be carried out each row name in corresponding relation type database successively simply by the order of A--Z) of HBase in self-defined Hadoop platform.In Hive, each row name of list structure is identical with the row name of list structure in relevant database, and self-defining row name is the row name corresponded in HBase herein.
So, if the major key in the tables of data of relevant database is Macintosh, then according to certain rule, splicing is carried out to Macintosh, as the row major key of HBase.Rule herein need be able to be carried out self-defined according to the specific of data and processing demands, as long as ensure that field that every bar record splices according to this rule uniquely.Such as, the field in combination major key is spliced by the mode that can directly connect with underscore.
Step 102, the Operation Log of relevant database described in Real-time Obtaining.
First, the identify label that this relevant database pre-configured is unique and initiation site.
In this step, except for except relevant database configuration unique identity and initiation site, other necessary informations can also be configured, as the IP address of relevant database place main frame, service port number, database user name and password etc.
Secondly, the Operation Log of Real-time Obtaining relevant database specifically comprises:
According to identify label and initiation site, from initiation site, obtain the Operation Log of relevant database corresponding to described identify label.
Finally, after the Operation Log of relevant database described in Real-time Obtaining, upgrade the initiation site of described relevant database; And according to the initiation site after renewal, obtain the next Operation Log of this relevant database.
It should be noted that, initiation site is the variable be kept in internal memory, successfully getting Operation Log after at every turn, just directly can be revised the value of this variable by program, namely upgrade this initiation site, this value be written in configuration file by program simultaneously.Be saved in file is after abnormal program termination, lose current initiation site to prevent.Like this, renewal progress when program exited last time is obtained when lower secondary program starts by this initiation site preserved in this file.
Site is a kind of station location marker of journalizing record in relevant database, and initiation site is then that station location marker starting to obtain Operation Log, and current site refers to daily record site current in relevant database.If be not configured initiation site, be then defaulted as the current site in relevant database.About the acquisition in site, illustrate, in Mysql database, obtain site by showmasterstatus.
More preferably, for each relevant database, a corresponding individual threads can be configured, receive the change data of the Operation Log of relevant database corresponding to identify label, and the change data received are stored in the message queue of a limited size in order; Or,
When not receiving the change data obtained in change request of data and message queue and exceeding threshold value, then the change data in described message queue are stored in corresponding catalogue file in order.
Initiate request of data once have like this, the corresponding change data be stored in corresponding catalogue file can preferentially be sent.
While the unique identify label of this relevant database pre-configured and initiation site, said method also comprises: relational data database data full dose imported in the database in this Hadoop platform by Sqoop.
It should be noted that, the identify label unique due to this relevant database pre-configured and initiation site, and this is also unique, so the process in the database only including and once relational data database data full dose imported in described Hadoop platform can be found out in this programme, import after Hadoop platform by data full dose, follow-up Data Update be in real time, increment but not the mode that general full dose again imports import in Hadoop platform.
Step 103, according to the Operation Log obtained, obtains the change data of described relevant database, according to the described mapping relations file set up, by the change Data Update of acquisition in the HBase of Hadoop.
Further, also the change data of acquisition are kept in local file, and record more new historical.
It should be noted that about the method, for relevant database, the main three kinds of operations paying close attention to it, namely increase, and upgrade and delete.And HBase only has increase data in essence, its renewal rewards theory and deletion action (not having update) are very alike, all renewal and deletion action is in itself all carry out in follow-up merging (Compact) process.
For this reason, the insertion of the renewal rewards theory of HBase is corresponding relevant database and upgrade two kinds of operations.For insertion, being mapped in HBase is then simple renewal rewards theory.Specifically,
First, for renewal, then to distinguish according to whether upgrading major key: if do not upgrade major key, be mapped in HBase and be still simple renewal rewards theory, otherwise, if having updated major key, then can be mapped to the multiple operations in HBase.Because HBase can preserve the historical data of multiple version according to timestamp, for this reason, when upgrading major key, then need first to take out corresponding numerical value by old master's key, then the data of new major key and old value composition are deposited in HBase database, then recycle new major key and new value, be updated in HBase, finally by old master's key, old value is deleted from HBase, like this, then still save the historical data of multiple version in HBase.
Finally, for deletion action, be mapped in HBase is still deletion action, but be only mark in HBase, real data deletion will be carried out in Compact process (it is noted herein that, after deletion action completes, these data are externally invisible, can not affect the Query Result of data).
Fig. 2 is the composition structural representation that the present invention realizes the device of real-time incremental synchrodata, be applied to the data importing of relevant database to distributed system architecture (Hadoop), as shown in Figure 2, comprising: build table module, log acquisition module, multiple daily record resolution client module and data update module.Wherein,
Build table module, for the list structure information according to relevant database, distributed, in the PostgreSQL database (HBase) of row, generate the mapping relations file corresponding with described relevant database.
Further, above-mentioned build table module also for:
If the major key in the tables of data of above-mentioned relation type database is Macintosh, then according to certain rule, splicing is carried out to Macintosh, as the row major key of above-mentioned HBase.
It should be noted that, above-mentionedly build table module, also for generating the contingency table corresponding with relevant database in Tool for Data Warehouse (Hive) with HBase.This contingency table refers to the tables of data consistent with list structure in relevant database created in Hive with HBase.After historical data imports to Hadoop platform, in relevant database, the renewal of tables of data can be updated in the correspondence table in Hive and HBase.HBase is used for data and stores, and Hive provides query function.This contingency table creates by Hive script.
Log acquisition module, for the Operation Log of relevant database described in Real-time Obtaining.
First, log acquisition module specifically for:
The identify label that this relevant database pre-configured is unique and initiation site;
According to identify label and initiation site, from initiation site, obtain the Operation Log of the corresponding relevant database of above-mentioned identify label.
In addition, log acquisition module also for:
Receive the data of the Operation Log of the corresponding relevant database of above-mentioned identify label, and the data received are stored in order in a message queue, for corresponding described daily record resolution client module acquisition request; Or,
If the data in described message queue when the data in message queue exceed threshold value, are then stored in corresponding catalogue file by data in order described in the non-acquisition request of described daily record resolution client module.
Finally, log acquisition module also for:
Upgrade the initiation site of this relevant database; According to the initiation site after renewal, obtain the next Operation Log of this relevant database.
Each daily record resolution client module, is connected with described log acquisition module respectively, for receiving the Operation Log of described log acquisition module transmission and changing data, and the change data of acquisition is sent to data update module.
Wherein, each daily record resolution client module, before the Operation Log receiving the transmission of described log acquisition module and change data, also for the unique identity of the relevant database according to correspondence and the annexation with log acquisition module, send to described log acquisition module and obtain request of data
Each daily record resolution client module also for: when data update module does not start, the change data received are stored in local file.
Data update module, for the change data that the described each daily record resolution client module of real-time reception sends, and according to the above-mentioned mapping relations file built table module and set up, by the change Data Update of acquisition in the HBase of distributed system architecture Hadoop.
Data update module also for: the change data of acquisition are kept in local file, and record more new historical.
In addition, said apparatus also comprises: full dose derivative module, for relational data database data full dose being imported in the HBase in Hadoop platform by Sqoop.
In this device, it should be noted that, the quantity that client end module is resolved in daily record is identical with the quantity of relevant database.
Embodiment one
This enforcement Sino-Israel relations type database Mysql is example, describes the real-time incremental Data Update how realizing HBase database in Mysql database to Hadoop in detail.
First, configuration target Mysql database is arranged, and opens Mysql binary log write-in functions, and is set to row pattern; Building in table module, configuration want synchronous Mysql database information, after having configured, table module is built in operation, in Hive with HBase, create contingency table corresponding with relevant database, generate the mapping relations file of tables of data in tables of data and HBase in the relevant database for data update module simultaneously; Have one to table info in hypothetical target database, its list structure is as follows:
| Field name | Field type | Explanation |
| id | bigint | From increasing major key |
| name | varchar(10) | |
| age | int |
The mapping relations file content generated is as follows:
{"COLUMN_FAMILY":"C","SQOOP_INFO":{"COLS":{"AGE":"C","ID":"A","NAME":"B"},"IS_PK_INT":true,"PK":["ID"]}}
Wherein, COLUMN_FAMILY is the row Praenomen in HBase, and there are three row in this row race, is distinguished called after A here, B, C, correspond respectively to the ID of info table, NAME, AGE, wherein the major key of info table is ID, and be Int type, in HBase, this tables of data name is called SQOOP_INFO.
Secondly, be the information such as the database configuration unique identity of target Mysql in relevant database log acquisition module, after completing, start relevant database log acquisition module in background process mode.The object starting relevant database log acquisition module before carrying out full dose derivative is, if target Mysql database data is many, need the cost regular hour relationship by objective (RBO) type database data all could be imported in Hadoop, and during this derivative, the data of target Mysql database may change, as inserted new data or renewal deletes partial data.So need to open relevant database log acquisition module before this to record Mysql Operation Log interior during this period, and after derivative completes, the data variation occurred during this is synchronized in the HBase database in Hadoop.
Finally, after startup relevant database log acquisition module, Sqoop is utilized to carry out full dose data derivative.After derivative completes, start relevant database daily record resolution client module and daily record update module in background process mode.Relevant database daily record resolution client module is that the database configuration unique identity of target Mysql and link information thereof are from relevant database log acquisition module request data by relevant database log acquisition module; Daily record update module obtains data by sending request to relevant database daily record resolution client module, and according to the mapped file generated in step one by the Data Update that obtains in the HBase database of Hadoop.
After all modules have been disposed, all operations to target Mysql database, all see corresponding data variation (time delay is substantially within one point of kind) by Hive inquiry or HBase inquiry, and corresponding Data Update historical record can be found from the daily record of data update module.Here be several in relevant database log acquisition module and relevant database daily record resolution client module, and the data layout sample transmitted between data update module:
{"COLUMN_NEWVALUE":["1","alex","25"],"COLUMN_OLDVALUE":["","",""],"OPERATETABLE":"info","PK_NEWVALUE":["1"],"PK_OLDVALUE":[""],"DATABASE":"sqoop","PK":["id"],"OPERATETYPE":"INSERT","USERNAME":"sqoop","COLUMN":["id","name","age"],"OPERATETIME":"2014-05-3003:20:40:000","TABLESPACE":""}
This is update, so its old value is all empty.Here is two renewal rewards theory, and Article 1 does not upgrade major key, and Article 2 have updated major key, and (this point can be found out from action type OPERATETYPE, and the value that have updated the OPERATETYPE of major key is defined as " UPDATE_FIELDCOMP_PK "; And the value not upgrading major key is defined as " UPDATE_FIELDCOMP "):
{"COLUMN_NEWVALUE":["tom"],"COLUMN_OLDVALUE":["alex"],"OPERATETABLE":"info","PK_NEWVALUE":["1"],"PK_OLDVALUE":["1"],"DATABASE":"sqoop","PK":["id"],"OPERATETYPE":"UPDATE_FIELDCOMP","USERNAME":"sqoop","COLUMN":["name"],"OPERATETIME":"2014-05-3003:21:28.000","TABLESPACE":""}
{"COLUMN_NEWVALUE":[],"COLUMN_OLDVALUE":[],"OPERATETABLE":"info","PK_NEWVALUE":["222"],"PK_OLDVALUE":["1"],"DATABASE":"sqoop","PK":["id"],"OPERATETYPE":"UPDATE_FIELDCOMP_PK","USERNAME":"sqoop","COLUMN":[],"OPERATETIME":"2014-05-3003:24:23.000","TABLESPACE":""}
Then, below, one is deletion action:
{"COLUMN_NEWVALUE":["",""],"COLUMN_OLDVALUE":["alex","25"],"OPERATETABLE":"info","PK_NEWVALUE":[""],"PK_OLDVALUE":["1"],"DATABASE":"sqoop","PK":["id"],"OPERATETYPE":"DELETE","USERNAME":"sqoop","COLUMN":["name","age"],"OPERATETIME":"2014-05-3003:29:39.000","TABLESPACE":""}
Now the key word of definition in above-mentioned four operations is described as follows:
COLUMN_NEWVALUE: list after this operation completes successively, the new value of each field that this row intermediate value changes;
COLUMN_OLDVALUE: list before this operation carries out successively, each old value that this line item intermediate value changes;
OPERATETABLE: which table representing operation;
PK_NEWVALUE: after this operation is carried out, the new value of this line item major key;
PK_OLDVALUE: before this operation is carried out, the old value of this line item major key;
DATABASE: the database-name in the Mysql of this operation correspondence;
PK: the row name of major key in this tables of data;
OPERATETYPE: action type, comprise insertion (INSERT), upgrade (UPDATE_FIELDCOMP and UPDATE_FIELDCOMP_PK, the former does not upgrade major key, the latter represent have updated major key), delete (DELETE);
USERNAME: the user name of database;
COLUMN: the title of each row that this tables of data intermediate value changes;
OPERATETIME: the time that operation occurs;
TABLESPACE: database table space.
The all or part of step that one of ordinary skill in the art will appreciate that in said method is carried out instruction related hardware by program and is completed, and described program can be stored in computer-readable recording medium, as ROM (read-only memory), disk or CD etc.Alternatively, all or part of step of above-described embodiment also can use one or more integrated circuit to realize.Correspondingly, each module/unit in above-described embodiment can adopt the form of hardware to realize, and the form of software function module also can be adopted to realize.The application is not restricted to the combination of the hardware and software of any particular form.
The above, be only preferred embodiments of the present invention, be not intended to limit protection scope of the present invention.Within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (11)
1. realize a method for real-time incremental synchrodata, be applied to the data importing of relevant database to distributed system architecture Hadoop, comprise:
According to the list structure information of relevant database, in the PostgreSQL database HBase towards row, generate the mapping relations file corresponding with described relevant database;
The Operation Log of relevant database described in Real-time Obtaining;
According to the Operation Log obtained, obtain the change data of described relevant database, according to the described mapping relations file set up, by the change Data Update of acquisition in the HBase of Hadoop.
2. method according to claim 1, is characterized in that, the identify label of pre-configured described relevant database and initiation site; The Operation Log of described Real-time Obtaining relevant database comprises:
According to described identify label and initiation site, from initiation site, obtain the Operation Log of relevant database corresponding to described identify label.
3. method according to claim 2, is characterized in that, the Operation Log of the described relevant database of described acquisition comprises:
Receive the change data of the Operation Log of relevant database corresponding to described identify label, and the change data received are stored in a message queue in order; Or,
When not receiving the change data obtained in change request of data and message queue and exceeding threshold value, then the change data in described message queue are stored in corresponding catalogue file in order.
4. the method according to claim 1 or 2 or 3, is characterized in that, after the Operation Log of described Real-time Obtaining relevant database, described method also comprises:
Upgrade the initiation site of described relevant database;
According to the initiation site after renewal, obtain the next Operation Log of described relevant database.
5. method according to claim 4, is characterized in that, described method also comprises: the described change data obtained be kept in local file, and record more new historical.
6. realize a device for real-time incremental synchrodata, be applied to the data importing of relevant database to distributed system architecture Hadoop, comprise: build table module, log acquisition module, multiple daily record resolution client module and data update module, wherein:
Build table module, for the list structure information according to relevant database, distributed, in the PostgreSQL database HBase of row, generate the mapping relations file corresponding with described relevant database;
Log acquisition module, for the Operation Log of relevant database described in Real-time Obtaining;
Each daily record resolution client module, is connected with described log acquisition module respectively, for receiving the Operation Log of described log analyzing module transmission and changing data, and the change data of acquisition is sent to data update module;
Data update module, for the change data that the described each daily record resolution client module of real-time reception sends, and according to the described described mapping relations file built table module and set up, by the change Data Update of acquisition in the HBase of Hadoop.
7. device according to claim 6, is characterized in that, described log acquisition module specifically for:
The identify label that pre-configured described relevant database is unique and initiation site;
According to described identify label and initiation site, from initiation site, obtain the Operation Log of the corresponding relevant database of described identify label.
8. device according to claim 6, is characterized in that, described log acquisition module also for:
Receive the data of the Operation Log of relevant database corresponding to described identify label, and the data received are stored in order in a message queue, for corresponding described daily record resolution client module acquisition request; Or,
If the data in described message queue when the data in message queue exceed threshold value, are then stored in corresponding catalogue file by data in order described in the non-acquisition request of described daily record resolution client module.
9. the device according to claim 6 or 7 or 8, is characterized in that, described log acquisition module also for:
Upgrade the initiation site of described relevant database;
According to the initiation site after renewal, obtain the next Operation Log of described relevant database.
10. device according to claim 9, is characterized in that, described daily record resolution client module also for: when described data update module does not start, the described change data received are stored in local file.
11. devices according to claim 9, is characterized in that, described data update module also for: the described change data obtained are kept in local file, and record more new historical.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410321182.2A CN105243067B (en) | 2014-07-07 | 2014-07-07 | A kind of method and device for realizing real-time incremental synchrodata |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410321182.2A CN105243067B (en) | 2014-07-07 | 2014-07-07 | A kind of method and device for realizing real-time incremental synchrodata |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105243067A true CN105243067A (en) | 2016-01-13 |
| CN105243067B CN105243067B (en) | 2019-06-28 |
Family
ID=55040718
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410321182.2A Active CN105243067B (en) | 2014-07-07 | 2014-07-07 | A kind of method and device for realizing real-time incremental synchrodata |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105243067B (en) |
Cited By (58)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN105847378A (en) * | 2016-04-13 | 2016-08-10 | 北京思特奇信息技术股份有限公司 | Big data synchronizing method and system |
| CN105956123A (en) * | 2016-05-03 | 2016-09-21 | 无锡雅座在线科技发展有限公司 | Local updating software-based data processing method and apparatus |
| CN106021422A (en) * | 2016-05-13 | 2016-10-12 | 北京思特奇信息技术股份有限公司 | Relational database-based method and system for forming Hive data warehouse |
| CN106126551A (en) * | 2016-06-13 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of generation method of Hbase database access daily record, Apparatus and system |
| CN106294713A (en) * | 2016-08-09 | 2017-01-04 | 深圳中兴网信科技有限公司 | The method of data synchronization resolved based on Incremental Log and data synchronization unit |
| CN106339408A (en) * | 2016-08-10 | 2017-01-18 | 深圳中兴网信科技有限公司 | Data synchronization method, data synchronization device and server |
| CN106407329A (en) * | 2016-09-05 | 2017-02-15 | 国网江苏省电力公司南通供电公司 | Method for automatically importing incremental data from massive platform to hadoop platform |
| CN106446243A (en) * | 2016-10-10 | 2017-02-22 | 山东浪潮云服务信息科技有限公司 | Data integration structure of relational database |
| CN106599061A (en) * | 2016-11-16 | 2017-04-26 | 成都九洲电子信息系统股份有限公司 | SQLite-based embedded database synchronization method |
| CN106682140A (en) * | 2016-12-20 | 2017-05-17 | 华北计算技术研究所（中国电子科技集团公司第十五研究所） | Multi-system user incremental synchronization method based on timestamps and mapping strategies |
| CN106682119A (en) * | 2016-12-08 | 2017-05-17 | 杭州销冠网络科技有限公司 | System and method for asynchronous data synchronization on basis of http service aspect and log system |
| CN106844682A (en) * | 2017-01-25 | 2017-06-13 | 北京百分点信息科技有限公司 | Method for interchanging data, apparatus and system |
| CN106874389A (en) * | 2017-01-11 | 2017-06-20 | 腾讯科技（深圳）有限公司 | The moving method and device of data |
| CN106960007A (en) * | 2017-02-28 | 2017-07-18 | 北京京东尚科信息技术有限公司 | A kind of data-updating method, device and electronic equipment |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107330003A (en) * | 2017-06-12 | 2017-11-07 | 上海藤榕网络科技有限公司 | Method of data synchronization, system, memory and data syn-chronization equipment |
| CN107320959A (en) * | 2017-06-28 | 2017-11-07 | 网易（杭州）网络有限公司 | Game role identification information generation method, device, medium and electronic equipment |
| WO2018001135A1 (en) * | 2016-06-30 | 2018-01-04 | 华为技术有限公司 | Method for processing database transaction, client and server |
| CN107741994A (en) * | 2017-11-09 | 2018-02-27 | 校脸科技（北京）有限公司 | A kind of data-updating method and device |
| CN107958082A (en) * | 2017-12-15 | 2018-04-24 | 杭州有赞科技有限公司 | Offline increment synchronization method and system of the database to data warehouse |
| CN107967279A (en) * | 2016-10-19 | 2018-04-27 | 北京国双科技有限公司 | The data-updating method and device of distributed data base |
| CN108009207A (en) * | 2017-11-06 | 2018-05-08 | 东软集团股份有限公司 | Incremental data inquiry method and device, storage medium, electronic equipment |
| WO2018090249A1 (en) * | 2016-11-16 | 2018-05-24 | Huawei Technologies Co., Ltd. | Log-structured storage method and server |
| CN108228755A (en) * | 2017-12-21 | 2018-06-29 | 江苏瑞中数据股份有限公司 | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method |
| CN108228397A (en) * | 2016-12-22 | 2018-06-29 | 深圳市优朋普乐传媒发展有限公司 | The method and apparatus that a kind of cluster span computer room synchronizes |
| CN108255838A (en) * | 2016-12-28 | 2018-07-06 | 航天信息股份有限公司 | A kind of method and system for establishing the intermediate data warehouse for big data analysis |
| CN108920698A (en) * | 2018-07-16 | 2018-11-30 | 北京京东金融科技控股有限公司 | A kind of method of data synchronization, device, system, medium and electronic equipment |
| CN109189852A (en) * | 2018-08-01 | 2019-01-11 | 武汉达梦数据库有限公司 | A kind of method that data are synchronous and the device synchronous for data |
| CN109213792A (en) * | 2018-07-06 | 2019-01-15 | 武汉斗鱼网络科技有限公司 | Method, server-side, client, device and the readable storage medium storing program for executing of data processing |
| CN109241184A (en) * | 2018-08-20 | 2019-01-18 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109241033A (en) * | 2018-08-21 | 2019-01-18 | 北京京东尚科信息技术有限公司 | The method and apparatus for creating real-time data warehouse |
| CN109582736A (en) * | 2018-11-22 | 2019-04-05 | 平安科技（深圳）有限公司 | Synchronous method, device and the computer equipment of loan transaction list table |
| CN109997125A (en) * | 2016-09-15 | 2019-07-09 | 英国天然气控股有限公司 | System for importing data to data storage bank |
| CN110175209A (en) * | 2019-04-12 | 2019-08-27 | 中国人民财产保险股份有限公司 | Incremental data synchronization method, system, equipment and storage medium |
| CN110362582A (en) * | 2018-04-03 | 2019-10-22 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for realizing zero shutdown upgrading |
| CN110413595A (en) * | 2019-06-28 | 2019-11-05 | 万翼科技有限公司 | A kind of data migration method and relevant apparatus applied to distributed data base |
| CN110727684A (en) * | 2019-10-08 | 2020-01-24 | 浪潮软件股份有限公司 | Incremental data synchronization method for big data statistical analysis |
| CN110737720A (en) * | 2019-09-06 | 2020-01-31 | 苏宁云计算有限公司 | DB2 database data synchronization method, device and system |
| CN110837535A (en) * | 2018-08-16 | 2020-02-25 | 中国移动通信集团江西有限公司 | Data synchronization method, device, equipment and medium |
| CN111221918A (en) * | 2019-11-04 | 2020-06-02 | 深圳力维智联技术有限公司 | Data updating method, device, product and medium based on relational database |
| CN111259178A (en) * | 2020-01-19 | 2020-06-09 | 罗普特科技集团股份有限公司 | Image data synchronization method, device and system |
| CN111339103A (en) * | 2020-03-13 | 2020-06-26 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full fragmentation and incremental log analysis |
| CN112000678A (en) * | 2020-08-20 | 2020-11-27 | 北京达佳互联信息技术有限公司 | Data synchronization method, device, server and storage medium |
| CN112115200A (en) * | 2020-09-16 | 2020-12-22 | 北京奇艺世纪科技有限公司 | Data synchronization method and device, electronic equipment and readable storage medium |
| WO2021003532A1 (en) * | 2019-07-09 | 2021-01-14 | Newsouth Innovations Pty Limited | Application and database migration to a block chain data lake system |
| CN112380180A (en) * | 2020-11-17 | 2021-02-19 | 平安普惠企业管理有限公司 | Data synchronization processing method, device, equipment and storage medium |
| CN112395360A (en) * | 2020-12-01 | 2021-02-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN112765180A (en) * | 2021-01-27 | 2021-05-07 | 上海英方软件股份有限公司 | Method and device for analyzing column names of table building logs of DB2 database |
| CN112800073A (en) * | 2021-01-27 | 2021-05-14 | 浪潮云信息技术股份公司 | Method for updating Delta Lake based on NiFi |
| CN112966046A (en) * | 2021-03-03 | 2021-06-15 | 北京金山云网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113377871A (en) * | 2021-06-22 | 2021-09-10 | 特赞（上海）信息科技有限公司 | Data synchronization method, device and storage medium |
| CN113407601A (en) * | 2020-03-17 | 2021-09-17 | 北京国双科技有限公司 | Data acquisition method and device, storage medium and electronic equipment |
| CN113495894A (en) * | 2020-04-01 | 2021-10-12 | 北京京东振世信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113515569A (en) * | 2020-04-09 | 2021-10-19 | 阿里巴巴集团控股有限公司 | Data synchronization method, device and system |
| CN114547199A (en) * | 2022-02-23 | 2022-05-27 | 阿维塔科技(重庆)有限公司 | Database increment synchronous response method and device and computer readable storage medium |
| CN115374199A (en) * | 2022-08-08 | 2022-11-22 | 广州小飞信息科技有限公司 | Big data based configuration type extensible statistical warehousing system and method thereof |
| WO2023109286A1 (en) * | 2021-12-17 | 2023-06-22 | 蚂蚁区块链科技(上海)有限公司 | Data synchronization method and apparatus |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102841897A (en) * | 2011-06-23 | 2012-12-26 | 阿里巴巴集团控股有限公司 | Incremental data extracting method, device and system |
| CN103631907A (en) * | 2013-11-26 | 2014-03-12 | 中国科学院信息工程研究所 | Method and system for migrating relational data to HBbase |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
-
2014
- 2014-07-07 CN CN201410321182.2A patent/CN105243067B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102841897A (en) * | 2011-06-23 | 2012-12-26 | 阿里巴巴集团控股有限公司 | Incremental data extracting method, device and system |
| US20130073516A1 (en) * | 2011-06-23 | 2013-03-21 | Alibaba Group Holding Limited | Extracting Incremental Data |
| CN103631907A (en) * | 2013-11-26 | 2014-03-12 | 中国科学院信息工程研究所 | Method and system for migrating relational data to HBbase |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
Non-Patent Citations (1)
| Title |
|---|
| 王玉标 等: ""异构环境下数据库增量同步更新机制"", 《计算机工程与设计》 * |
Cited By (81)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN105824744B (en) * | 2016-03-21 | 2018-06-15 | 焦点科技股份有限公司 | A kind of real-time logs capturing analysis method based on B2B platform |
| CN105847378A (en) * | 2016-04-13 | 2016-08-10 | 北京思特奇信息技术股份有限公司 | Big data synchronizing method and system |
| CN105847378B (en) * | 2016-04-13 | 2019-06-28 | 北京思特奇信息技术股份有限公司 | A kind of method and system for realizing that big data is synchronous |
| CN105956123A (en) * | 2016-05-03 | 2016-09-21 | 无锡雅座在线科技发展有限公司 | Local updating software-based data processing method and apparatus |
| CN106021422A (en) * | 2016-05-13 | 2016-10-12 | 北京思特奇信息技术股份有限公司 | Relational database-based method and system for forming Hive data warehouse |
| CN106021422B (en) * | 2016-05-13 | 2019-04-09 | 北京思特奇信息技术股份有限公司 | A kind of method and system forming Hive data warehouse based on relevant database |
| CN106126551A (en) * | 2016-06-13 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of generation method of Hbase database access daily record, Apparatus and system |
| WO2018001135A1 (en) * | 2016-06-30 | 2018-01-04 | 华为技术有限公司 | Method for processing database transaction, client and server |
| CN106294713A (en) * | 2016-08-09 | 2017-01-04 | 深圳中兴网信科技有限公司 | The method of data synchronization resolved based on Incremental Log and data synchronization unit |
| CN106339408A (en) * | 2016-08-10 | 2017-01-18 | 深圳中兴网信科技有限公司 | Data synchronization method, data synchronization device and server |
| CN106339408B (en) * | 2016-08-10 | 2019-08-23 | 深圳中兴网信科技有限公司 | Method of data synchronization, data synchronization unit and server |
| CN106407329A (en) * | 2016-09-05 | 2017-02-15 | 国网江苏省电力公司南通供电公司 | Method for automatically importing incremental data from massive platform to hadoop platform |
| CN106407329B (en) * | 2016-09-05 | 2019-06-25 | 国网江苏省电力公司南通供电公司 | Magnanimity platform automates the method for importing incremental data toward hadoop platform |
| CN109997125A (en) * | 2016-09-15 | 2019-07-09 | 英国天然气控股有限公司 | System for importing data to data storage bank |
| US11461294B2 (en) | 2016-09-15 | 2022-10-04 | Hitachi Vantara Llc | System for importing data into a data repository |
| CN106446243A (en) * | 2016-10-10 | 2017-02-22 | 山东浪潮云服务信息科技有限公司 | Data integration structure of relational database |
| CN107967279A (en) * | 2016-10-19 | 2018-04-27 | 北京国双科技有限公司 | The data-updating method and device of distributed data base |
| CN106599061A (en) * | 2016-11-16 | 2017-04-26 | 成都九洲电子信息系统股份有限公司 | SQLite-based embedded database synchronization method |
| US10430341B2 (en) | 2016-11-16 | 2019-10-01 | Huawei Technologies Co., Ltd. | Log-structured storage method and server |
| WO2018090249A1 (en) * | 2016-11-16 | 2018-05-24 | Huawei Technologies Co., Ltd. | Log-structured storage method and server |
| CN106599061B (en) * | 2016-11-16 | 2020-06-30 | 成都九洲电子信息系统股份有限公司 | SQLite-based embedded database synchronization method |
| CN106682119A (en) * | 2016-12-08 | 2017-05-17 | 杭州销冠网络科技有限公司 | System and method for asynchronous data synchronization on basis of http service aspect and log system |
| CN106682140A (en) * | 2016-12-20 | 2017-05-17 | 华北计算技术研究所（中国电子科技集团公司第十五研究所） | Multi-system user incremental synchronization method based on timestamps and mapping strategies |
| CN108228397A (en) * | 2016-12-22 | 2018-06-29 | 深圳市优朋普乐传媒发展有限公司 | The method and apparatus that a kind of cluster span computer room synchronizes |
| CN108255838A (en) * | 2016-12-28 | 2018-07-06 | 航天信息股份有限公司 | A kind of method and system for establishing the intermediate data warehouse for big data analysis |
| CN108255838B (en) * | 2016-12-28 | 2022-02-18 | 航天信息股份有限公司 | Method and system for establishing intermediate data warehouse for big data analysis |
| CN106874389A (en) * | 2017-01-11 | 2017-06-20 | 腾讯科技（深圳）有限公司 | The moving method and device of data |
| CN106844682B (en) * | 2017-01-25 | 2019-08-16 | 北京百分点信息科技有限公司 | Method for interchanging data, apparatus and system |
| CN106844682A (en) * | 2017-01-25 | 2017-06-13 | 北京百分点信息科技有限公司 | Method for interchanging data, apparatus and system |
| CN106960007A (en) * | 2017-02-28 | 2017-07-18 | 北京京东尚科信息技术有限公司 | A kind of data-updating method, device and electronic equipment |
| CN107330003A (en) * | 2017-06-12 | 2017-11-07 | 上海藤榕网络科技有限公司 | Method of data synchronization, system, memory and data syn-chronization equipment |
| CN107320959A (en) * | 2017-06-28 | 2017-11-07 | 网易（杭州）网络有限公司 | Game role identification information generation method, device, medium and electronic equipment |
| CN107320959B (en) * | 2017-06-28 | 2020-10-23 | 网易（杭州）网络有限公司 | Game role identification information generation method, device, medium and electronic equipment |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN108009207A (en) * | 2017-11-06 | 2018-05-08 | 东软集团股份有限公司 | Incremental data inquiry method and device, storage medium, electronic equipment |
| CN107741994A (en) * | 2017-11-09 | 2018-02-27 | 校脸科技（北京）有限公司 | A kind of data-updating method and device |
| CN107958082B (en) * | 2017-12-15 | 2021-03-26 | 杭州有赞科技有限公司 | Off-line increment synchronization method and system from database to data warehouse |
| CN107958082A (en) * | 2017-12-15 | 2018-04-24 | 杭州有赞科技有限公司 | Offline increment synchronization method and system of the database to data warehouse |
| CN108228755A (en) * | 2017-12-21 | 2018-06-29 | 江苏瑞中数据股份有限公司 | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method |
| CN110362582A (en) * | 2018-04-03 | 2019-10-22 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for realizing zero shutdown upgrading |
| CN109213792A (en) * | 2018-07-06 | 2019-01-15 | 武汉斗鱼网络科技有限公司 | Method, server-side, client, device and the readable storage medium storing program for executing of data processing |
| CN108920698A (en) * | 2018-07-16 | 2018-11-30 | 北京京东金融科技控股有限公司 | A kind of method of data synchronization, device, system, medium and electronic equipment |
| CN109189852A (en) * | 2018-08-01 | 2019-01-11 | 武汉达梦数据库有限公司 | A kind of method that data are synchronous and the device synchronous for data |
| CN109189852B (en) * | 2018-08-01 | 2021-05-28 | 武汉达梦数据库有限公司 | Data synchronization method and device for data synchronization |
| CN110837535A (en) * | 2018-08-16 | 2020-02-25 | 中国移动通信集团江西有限公司 | Data synchronization method, device, equipment and medium |
| CN109241184B (en) * | 2018-08-20 | 2024-03-15 | 中国平安人寿保险股份有限公司 | Data synchronization method, device, computer equipment and storage medium |
| CN109241184A (en) * | 2018-08-20 | 2019-01-18 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109241033A (en) * | 2018-08-21 | 2019-01-18 | 北京京东尚科信息技术有限公司 | The method and apparatus for creating real-time data warehouse |
| CN109582736A (en) * | 2018-11-22 | 2019-04-05 | 平安科技（深圳）有限公司 | Synchronous method, device and the computer equipment of loan transaction list table |
| CN110175209A (en) * | 2019-04-12 | 2019-08-27 | 中国人民财产保险股份有限公司 | Incremental data synchronization method, system, equipment and storage medium |
| CN110413595A (en) * | 2019-06-28 | 2019-11-05 | 万翼科技有限公司 | A kind of data migration method and relevant apparatus applied to distributed data base |
| GB2600315A (en) * | 2019-07-09 | 2022-04-27 | Newsouth Innovations Pty Ltd | Application and database migration to a block chain data lake system |
| WO2021003532A1 (en) * | 2019-07-09 | 2021-01-14 | Newsouth Innovations Pty Limited | Application and database migration to a block chain data lake system |
| CN110737720A (en) * | 2019-09-06 | 2020-01-31 | 苏宁云计算有限公司 | DB2 database data synchronization method, device and system |
| CN110727684B (en) * | 2019-10-08 | 2023-07-25 | 浪潮软件股份有限公司 | Incremental data synchronization method for big data statistical analysis |
| CN110727684A (en) * | 2019-10-08 | 2020-01-24 | 浪潮软件股份有限公司 | Incremental data synchronization method for big data statistical analysis |
| CN111221918A (en) * | 2019-11-04 | 2020-06-02 | 深圳力维智联技术有限公司 | Data updating method, device, product and medium based on relational database |
| CN111259178A (en) * | 2020-01-19 | 2020-06-09 | 罗普特科技集团股份有限公司 | Image data synchronization method, device and system |
| CN111339103A (en) * | 2020-03-13 | 2020-06-26 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full fragmentation and incremental log analysis |
| CN111339103B (en) * | 2020-03-13 | 2023-06-20 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full-quantity fragmentation and incremental log analysis |
| CN113407601A (en) * | 2020-03-17 | 2021-09-17 | 北京国双科技有限公司 | Data acquisition method and device, storage medium and electronic equipment |
| CN113495894A (en) * | 2020-04-01 | 2021-10-12 | 北京京东振世信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113515569B (en) * | 2020-04-09 | 2023-12-26 | 阿里巴巴集团控股有限公司 | Data synchronization method, device and system |
| CN113515569A (en) * | 2020-04-09 | 2021-10-19 | 阿里巴巴集团控股有限公司 | Data synchronization method, device and system |
| CN112000678A (en) * | 2020-08-20 | 2020-11-27 | 北京达佳互联信息技术有限公司 | Data synchronization method, device, server and storage medium |
| CN112115200B (en) * | 2020-09-16 | 2023-08-29 | 北京奇艺世纪科技有限公司 | Data synchronization method, device, electronic equipment and readable storage medium |
| CN112115200A (en) * | 2020-09-16 | 2020-12-22 | 北京奇艺世纪科技有限公司 | Data synchronization method and device, electronic equipment and readable storage medium |
| CN112380180A (en) * | 2020-11-17 | 2021-02-19 | 平安普惠企业管理有限公司 | Data synchronization processing method, device, equipment and storage medium |
| CN112395360A (en) * | 2020-12-01 | 2021-02-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN112395360B (en) * | 2020-12-01 | 2023-06-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN112800073A (en) * | 2021-01-27 | 2021-05-14 | 浪潮云信息技术股份公司 | Method for updating Delta Lake based on NiFi |
| CN112800073B (en) * | 2021-01-27 | 2023-03-28 | 浪潮云信息技术股份公司 | Method for updating Delta Lake based on NiFi |
| CN112765180A (en) * | 2021-01-27 | 2021-05-07 | 上海英方软件股份有限公司 | Method and device for analyzing column names of table building logs of DB2 database |
| CN112966046A (en) * | 2021-03-03 | 2021-06-15 | 北京金山云网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN112966046B (en) * | 2021-03-03 | 2024-04-12 | 北京金山云网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113377871A (en) * | 2021-06-22 | 2021-09-10 | 特赞（上海）信息科技有限公司 | Data synchronization method, device and storage medium |
| CN113377871B (en) * | 2021-06-22 | 2024-03-22 | 特赞（上海）信息科技有限公司 | Data synchronization method, device and storage medium |
| WO2023109286A1 (en) * | 2021-12-17 | 2023-06-22 | 蚂蚁区块链科技(上海)有限公司 | Data synchronization method and apparatus |
| CN114547199A (en) * | 2022-02-23 | 2022-05-27 | 阿维塔科技(重庆)有限公司 | Database increment synchronous response method and device and computer readable storage medium |
| CN115374199A (en) * | 2022-08-08 | 2022-11-22 | 广州小飞信息科技有限公司 | Big data based configuration type extensible statistical warehousing system and method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105243067B (en) | 2019-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105243067A (en) | Method and apparatus for realizing real-time increment synchronization of data | |
| CN108228814B (en) | Data synchronization method and device | |
| CN104090889B (en) | Data processing method and system | |
| CN101098248B (en) | Method and system for implementing universal network management based on configuration describing document | |
| CN107203395A (en) | Resource graded method, device and computer-readable recording medium and electronic equipment | |
| CN103916482A (en) | Data synchronous transmission method based on sqlite | |
| CN103164525B (en) | WEB application dissemination method and device | |
| CN101770515A (en) | Data block comparison based data updating method | |
| CN104778057A (en) | Terminal system upgrading method and device, and server | |
| CN102202087B (en) | Method for identifying storage equipment and system thereof | |
| CN107977396B (en) | Method and device for updating data table of KeyValue database | |
| CN104252452A (en) | Data management method and device | |
| CN104134109A (en) | Engineering configuration method and engineering configuration system integrating version management | |
| CN112217656B (en) | Method and device for synchronizing configuration information of network equipment in SD-WAN (secure digital-to-Wide area network) system | |
| CN104346479A (en) | Database synchronization method and database synchronization device | |
| CN103034735A (en) | Big data distributed file export method | |
| CN111666326A (en) | ETL scheduling method and device | |
| CN103699548A (en) | Method and equipment for recovering database data by using logs | |
| CN110895472A (en) | Method and device for identifying service change | |
| CN112434059A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN104378234A (en) | Cross-data-center data transmission processing method and system | |
| CN102917057A (en) | Individualized application pushing system and method | |
| CN105138927A (en) | Privacy data protection method and apparatus | |
| CN102375890A (en) | Data synchronization method for source terminal table of database without major key | |
| CN110781197A (en) | Hive offline synchronous verification method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Yang Wei Inventor after: Bai Junwei Inventor after: Wang Xiaofeng Inventor after: Feng Shicong Inventor before: Yang Wei Inventor before: Bai Junwei Inventor before: Wang Xiaofeng Inventor before: Feng Shicong Inventor before: Wu Minghui |
|
| COR | Change of bibliographic data | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220628 Address after: 15, second floor, east side of clean coal workshop, No. 68, Shijingshan Road, Shijingshan District, Beijing 100043 (cluster registration) Patentee after: Beijing Zhizhi Heshu Technology Co.,Ltd. Address before: 100193 No.310, building 4, courtyard 8, Dongbeiwang West Road, Haidian District, Beijing Patentee before: MININGLAMP SOFTWARE SYSTEMS Co.,Ltd. |
|
| TR01 | Transfer of patent right |