CN102999629B - A kind of relational database is to the asynchronous converting system of non-mode database and method - Google Patents
A kind of relational database is to the asynchronous converting system of non-mode database and method Download PDFInfo
- Publication number
- CN102999629B CN102999629B CN201210536616.1A CN201210536616A CN102999629B CN 102999629 B CN102999629 B CN 102999629B CN 201210536616 A CN201210536616 A CN 201210536616A CN 102999629 B CN102999629 B CN 102999629B
- Authority
- CN
- China
- Prior art keywords
- database
- data
- mode
- cell
- relational database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention discloses a kind of relational database to the asynchronous converting system of non-mode database and method, comprise relational database, described relational database by Signal transmissions to daily record audiomonitor, described daily record audiomonitor by Signal transmissions to mode regulator, described mode regulator by Signal transmissions to non-mode database, described daily record audiomonitor also by Signal transmissions to cell status register, described cell status register by Signal transmissions to cell state write-back device, described cell state write-back device by Signal transmissions to non-mode database.The present invention under not interact relation database performance prerequisite by the pattern of relational database and data asynchronous to being transformed into non-mode database by this device, asynchronous system can not cause relational database to block, based on relational database, non-mode database is auxiliary, for solving the I/O bottleneck problem of traditional relational when large data query.
Description
Technical field
The present invention relates to internet industry distributed data base field, particularly relate to a kind of relational database to the asynchronous converting system of non-mode database and method.
Background technology
Traditional relational is the principal mode of access data during current Web applies.But along with the development of Web2.0, the large technology such as data, cloud application, traditional relational database encounters bottleneck in I/O, extensibility, and when large data access and access, efficiency sharply declines, and also encounters a difficulty in enhanced scalability and high availability.In order to head it off, many schemes propose in succession, such as distributed data base, master-slave database, non-mode database.Non-mode database, be a kind of pattern database freely, its pattern is mainly the various ways such as relation, tree, figure, document.In fact, be difficult to abandon existing relational database completely and directly use non-mode database.Based on relational database, non-mode database is auxiliary, has become industry member and the generally accepted scheme of industrial community.
In existing technology, there is the method that two class relational databases and non-mode database carry out combining: (1) is with the method for user's plug-in unit, by non-mode geo-database integration in relational database, make relational database support relational operation and non-mode operation simultaneously.The method shortcoming is that the operation of a large amount of opening/closing table is very consuming time, reduces database performance; (2) relational database used in combination, based on relational database, non-mode database is auxiliary, uses relational database to carry out large Data Update operation, adopts certain mechanism by relational data base schema and data syn-chronization to non-mode database.Inquiry for large data directly uses non-mode database, and renewal still adopts relational database.The method advantage is the inquiry bottleneck problem both having solved large data in relational database, solves again non-mode database update bottleneck problem.The difficult point of the method how relational data base schema and data is transformed into non-mode database.The solution of prior art and realization mainly contain two kinds of methods: (1) by relational database trigger, by the change records of data to temporary query table.This question blank is made up of ID, table name, line number, timestamp and operation isotype usually.Then by the data syn-chronization of temporary query table to target non-mode database; (2) method (1) being improved, cancel temporary query table, by increasing the self-defined HTTP function of relational database, directly relational database data variation increment being sent to target non-mode database by http protocol, realize the synchronous of data and pattern.
To sum up, existing relational database mainly contains following problem to the conversion method of non-mode database: no matter (1) is trigger or User-Defined Functions, and it has a strong impact on relational database performance; (2) data transmission procedure cannot ensure the order of data manipulation, easily causes the inconsistent of data; (3) method of synchronization meeting blocking data storehouse process, reduces pattern and the Data Update performance of relational database.
Summary of the invention
Object of the present invention is exactly to solve the problem, there is provided a kind of relational database to the asynchronous converting system of non-mode database and method, the pattern of relational database and data are transformed into non-mode database by this device is asynchronous, based on relational database, non-mode database is auxiliary, for solving the I/O bottleneck problem of traditional relational when large data query.
To achieve these goals, the present invention adopts following technical scheme:
A kind of relational database is to the asynchronous converting system of non-mode database, comprise relational database, described relational database by Signal transmissions to daily record audiomonitor, described daily record audiomonitor by Signal transmissions to mode regulator, described mode regulator by Signal transmissions to non-mode database, described daily record audiomonitor also by Signal transmissions to cell status register, described cell status register by Signal transmissions to cell state write-back device, described cell state write-back device by Signal transmissions to non-mode database.
Described relational database is the source of relational database to the asynchronous converting system of non-mode database.
Described non-mode database is the target of relational database to the asynchronous converting system of non-mode database.
Described daily record audiomonitor is for monitoring the daily record of relational database.Relational database binary log based on row comprises schema update and Data Update, and described schema update records the SQL statement of each schema modification, and described Data Update records in each affairs with the data before and after behavior unit change.For update, before supervising audible change, data are NULL, and after change, data are newly-increased data; For renewal rewards theory, monitor and obtain data before and after amendment, for deletion action, before supervising audible change, data are deleted data, and after change, data are NULL.
The delta data based on row that described cell status register is responsible for daily record audiomonitor to listen to is split as cell state.
Described mode regulator is responsible for the patterns of change that daily record audiomonitor listens to be mapped to non-mode database.Mapping ruler is as follows: the one-one relationship in relational database is mapped as the relational structure in non-mode database, one-to-many in relational database or many-to-one relationship are mapped as the tree in non-mode database, and the many-to-many relationship in relational database is mapped as the pattern structure of non-mode database.Relational database database name is mapped as non-mode database name, and the table name of relational database is mapped as non-mode database collection name (Collection), and the column index of relational database is mapped as the column index (Index) of non-mode database.
Described cell state write-back device is responsible for the cell state key assignments ordered pair in cell status register to be written to non-mode database.
The conversion method that above-mentioned relation database adopts to the asynchronous converting system of non-mode database, specific works step is as follows:
Step (1): daily record audiomonitor listens to the relational database binary log based on row, and resolves, obtains the data that change and pattern, if what listen to is delta data, goes to step (2), otherwise goes to step (3).
Step (2): the Data Division before and after the change based on row is cell state by daily record audiomonitor, if cell data change, value in amendment unit trellis state key assignments ordered pair V and revision version, if cell data do not change, then do not carry out any process, according to said method cell state is stored in cell status register, goes to step (1).
All cell states in cell status register are written back to non-mode database, go to step (4) by step (3): daily record audiomonitor is resolved the patterns of change SQL statement listened to.
4) according to the schema modification SQL statement that daily record audiomonitor is monitored, mode regulator, by source relational data base schema modifications target non-mode database schema, goes to step (1).
The resolving of described step (3) is divided into following four steps:
Step (3-1): if cell state basic version is 1 and revision version is 0, illustrates that newly-increased cell is deleted, is left intact;
Step (3-2): if cell state basic version is greater than 1 and revision version is 0, deleted after newly-increased cell amendment is described, need to delete corresponding data in non-mode database;
Step (3-3): if cell state basic version is 1 and revision version is not 0, be illustrated as newly-increased cell, need cell state to be written back to non-mode database;
Step (3-4): if cell state basic version is greater than 0 and revision version is not 0, illustrate that cell is modified, needs to revise non-mode database corresponding data.
Above-mentioned relation database relates to following term to the asynchronous converting system of non-mode database:
Described cell refers to the data in the relational database bivariate table in the definition of cell state.
The data cells that described relational database is made up of ranks is formed.According to ranks positioning unit lattice, be expert at by major key key assignments ordered pair location database record, by row name locator data record column.Each cell state defines a key assignments ordered pair K → V, wherein K is defined as tlv triple < table name, major key key assignments ordered pair, row name >, and V is defined as two tuple < train values, revision version >.Revision version records the version information of these cell data, can represent by positive integer.Newly increasing data initial version is 1, and after each cell data change, version adds 1.After cell is deleted, version modify is 0.
Described cell state revision version is cell state latest edition in both the buffers.
Described cell state basic version is cell state initial revision version in both the buffers.
For newly increasing data, be split as cell state, its revision version and basic version are 1; For Update Table, be split as cell state, if this cell is not in cell status register, its revision version and basic version are 2, if this cell is in cell status register, its revision version is that former revision version adds 1, and basic version is former cell state basic version; For deletion data, revision version is 0.If cell is in cell status register, basic version is former cell basic version, if cell is not in cell status register, basic version is 0.
Beneficial effect of the present invention:
1 without the need to interact relation database performance just can implementation relation database to the asynchronous translation function of non-mode database;
2 asynchronous conversion equipments can ensure the order of data in data transmission procedure;
The obstruction of relational database process can not be caused in 3 asynchronous data transfer processes;
4 by the pattern of relational database and data asynchronous to being transformed into non-mode database by this device, based on relational database, non-mode database is auxiliary, for solving the I/O bottleneck problem of traditional relational when large data query.
Accompanying drawing explanation
Fig. 1 relational database is to the Organization Chart of the asynchronous conversion method of non-mode database and device;
Fig. 2 removes the process flow diagram of amending unit trellis state according to relational database delta data;
Fig. 3 cell state set is written back to the process flow diagram of non-mode database;
Wherein, 101, relational database, 102, daily record audiomonitor, 103, mode regulator, 104, cell status register, 105, cell state write-back device, 106, non-mode database.
Embodiment
Below in conjunction with accompanying drawing and embodiment, the invention will be further described.
As shown in Figure 1, a kind of relational database is to the asynchronous converting system of non-mode database, comprise relational database 101, described relational database 101 by Signal transmissions to daily record audiomonitor 102, described daily record audiomonitor 102 by Signal transmissions to mode regulator 103, described mode regulator 103 by Signal transmissions to non-mode database 106, described daily record audiomonitor 102 also by Signal transmissions to cell status register 104, described cell status register 104 by Signal transmissions to cell state write-back device 105, described cell state write-back device 105 by Signal transmissions to non-mode database 106.
Described relational database 101 is relational database 101 sources to the asynchronous conversion equipment of non-mode database 106.
Described non-mode database 106 is relational database 101 targets to the asynchronous conversion equipment of non-mode database 106.
Described daily record audiomonitor 102 is for monitoring the daily record of relational database 101.Relational database 101 binary log based on row comprises schema update and Data Update, and described schema update records the SQL statement of each schema modification, and described Data Update records in each affairs with the data before and after behavior unit change.For update, before supervising audible change, data are NULL, and after change, data are newly-increased data; For renewal rewards theory, monitor and obtain data before and after amendment, for deletion action, before supervising audible change, data are deleted data, and after change, data are NULL.
The delta data based on row that described cell status register 104 is responsible for daily record audiomonitor 102 to listen to is split as cell.
Described mode regulator 103 is responsible for the patterns of change that daily record audiomonitor 102 listens to be mapped to non-mode database 106.Mapping ruler is as follows: the one-one relationship in relational database 101 is mapped as the relational structure in non-mode database 106, one-to-many in relational database 101 or many-to-one relationship are mapped as the tree in non-mode database 106, and the many-to-many relationship in relational database 101 is mapped as the pattern structure of non-mode database 106.The database name of relational database 101 is mapped as the database name of non-mode database 106, the table name of relational database 101 is mapped as the set name (Collection) of non-mode database 106, and the column index of relational database 101 is mapped as the column index (Index) of non-mode database 106.
Described cell state write-back device 105 is responsible for the cell state key assignments ordered pair in cell status register 104 to be written to non-mode database 106.
As shown in Figure 2, remove the process flow diagram of the cell state in amending unit lattice status register 104 according to relational database 101 delta data, give cell state and generate specific works process.
Step (110): the database binary log that relational database 101 produces monitored by daily record audiomonitor 102, parses the delta data based on row and schema modification SQL statement;
Step (111): judge that resolving log information is data variation or patterns of change, if data variation enters step (112), if patterns of change enters step (126);
Step (112): if log information is data variation, continues judging unit lattice status register 104 usage space and whether reaches threshold value, just enter step (126) if reach; Step (113) is just entered according to the cell state in delta data amending unit lattice status register 104 if do not reach;
Step (113): judge the data variation type that daily record audiomonitor 102 parses; If change type just enters step (114) for increasing; If change type just enters step (117) for revising; If change type just enters step (122) for deleting;
Step (114): data variation type is for increasing: before change, data are NULL, after change, data are newly-increased data; Enter step (115);
Step (115): every bar is increased data and is split as multiple cell state by row, cell state is kept at cell status register 104; Enter step (116);
Step (116): basic version and the revision version of each cell state are set to 1; Enter step (128);
Step (117): data variation type is amendment: before change, data are data before amendment, after change, data are data after amendment;
Step (118): data after the amendment of every bar are split as cell state by row, only the cell state that changes of processing costs;
Step (119): judging unit trellis state whether Already in cell status register 104; If just entering step (120); If just do not entering step (121);
Step (120): if cell state Already in cell status register 104, then revise the value of this cell state, cell state revision version is added 1; Enter step (128);
Step (121): if cell state is not at cell status register 104, then revise the value of this cell state, cell state basic version and revision version are set to 2, be kept at cell status register 104; Enter step (128);
Step (122): data variation type is for deleting: before change, data are data before deleting, and after change, data are NULL; Enter step (123);
Step (123): by often row delete data be split as cell state by row, be kept at cell status register 104; Enter step (124);
Step (124): cell state value is set to NULL; Enter step (125);
Step (125): cell state revision version is set to 0; Enter step (128);
Step (126): if log information is patterns of change, uses cell state write-back device 105 that all cell states in cell status register 104 are written back to non-mode database 106; Enter step (127);
Step (127): the patterns of change SQL statement parsed according to daily record audiomonitor 102, using forestland adjuster 103 goes the pattern of revise goal non-mode database 106.
Step (128): judge whether to terminate, if just terminate, just return step (111) if not.
The concrete steps of described step (126) are as shown in figure (3), and Fig. 3 is the process flow diagram that cell state set in cell status register 104 is written back to non-mode database 106,
Step (130): take out a cell state from cell status register 104; Enter step (131);
Step (131): judge this cell state basic version; If basic version is 1, enter step (132); Otherwise enter step (137);
Step (132): if this cell state basic version is 1, illustrate that this cell is initially newly-increased data in cell status register 104;
Step (133): judge whether this cell state revision version is 0; If revision version is 0 just enter step (134); Otherwise enter step (135);
Step (134): if cell state revision version is 0, illustrates that this cell is for deleted after newly-increased data, is left intact; Enter step (143);
Step (135): if cell state revision version is not 0, illustrates that cell is for newly-increased data; Enter step (136);
Step (136): use cell state write-back device 105 using this cell state as newly-increased write back data to non-mode database 106; Enter step (143);
Step (137): if this cell state basic version is greater than 1, illustrate that this cell is initially Update Table in cell status register 104; Enter step (138);
Step (138): judge this cell state revision version; If 0 just enters step (139); Otherwise enter step (141);
Step (139): if cell state revision version is 0, deleted after the amendment of this cell is described; Enter step (140);
Step (140): use cell state write-back device 105 data item corresponding in non-mode database 106 for this cell to be deleted, enter step (143);
Step (141): if cell state revision version is not 0, illustrates that cell is Update Table; Enter step (142);
Step (142): use cell state write-back device 105 by the value of data item corresponding in this cell state last look correction non-mode database 106; Enter step (143);
Step (143): judge whether to terminate, if just terminate, just return step (130) if not.
By reference to the accompanying drawings the specific embodiment of the present invention is described although above-mentioned; but not limiting the scope of the invention; one of ordinary skill in the art should be understood that; on the basis of technical scheme of the present invention, those skilled in the art do not need to pay various amendment or distortion that creative work can make still within protection scope of the present invention.
Claims (1)
1. the conversion method that adopts to the asynchronous converting system of non-mode database of relational database, is characterized in that,
Described asynchronous converting system comprises relational database, described relational database by Signal transmissions to daily record audiomonitor, described daily record audiomonitor by Signal transmissions to mode regulator, described mode regulator by Signal transmissions to non-mode database, described daily record audiomonitor also by Signal transmissions to cell status register, described cell status register by Signal transmissions to cell state write-back device, described cell state write-back device by Signal transmissions to non-mode database, described relational database is the source of relational database to the asynchronous converting system of non-mode database, described non-mode database is the target of relational database to the asynchronous converting system of non-mode database, described daily record audiomonitor is for monitoring the daily record of relational database, relational database binary log based on row comprises schema update and Data Update, and described schema update records the SQL statement of each schema modification, and described Data Update records in each affairs with the data before and after behavior unit change, for update, before supervising audible change, data are NULL, and after change, data are newly-increased data, for renewal rewards theory, monitoring obtains data before and after amendment, for deletion action, before supervising audible change, data are deleted data, after change, data are NULL, the delta data based on row that described cell status register is responsible for daily record audiomonitor to listen to is split as cell state, and described mode regulator is responsible for the patterns of change that daily record audiomonitor listens to be mapped to non-mode database, mapping ruler is as follows: the one-one relationship in relational database is mapped as the relational structure in non-mode database, one-to-many in relational database or many-to-one relationship are mapped as the tree in non-mode database, and the many-to-many relationship in relational database is mapped as the pattern structure of non-mode database, relational database database name is mapped as non-mode database name, the table name of relational database is mapped as non-mode database collection name, the column index of relational database is mapped as the column index of non-mode database, and described cell state write-back device is responsible for the cell state key assignments ordered pair in cell status register to be written to non-mode database,
The specific works step of described conversion method is as follows:
Step (1): daily record audiomonitor listens to the relational database binary log based on row, and resolves, obtains the data that change and pattern, if what listen to is delta data, goes to step (2), otherwise goes to step (3);
Step (2): the delta data based on row is split as cell state by cell status register, if cell data change, value in amendment unit trellis state key assignments ordered pair and revision version, if cell data do not change, then do not carry out any process, according to said method cell state is stored in cell status register, goes to step (1);
All cell states in cell status register are written back to non-mode database, go to step (4) by step (3): daily record audiomonitor is resolved the patterns of change SQL statement listened to;
The resolving of described step (3) is divided into following four steps:
Step (3-1): if cell state basic version is 1 and revision version is 0, illustrates that newly-increased cell is deleted, is left intact;
Step (3-2): if cell state basic version is greater than 1 and revision version is 0, deleted after newly-increased cell amendment is described, need to delete corresponding data in non-mode database;
Step (3-3): if cell state basic version is 1 and revision version is not 0, be illustrated as newly-increased cell, need cell state to be written back to non-mode database;
Step (3-4): if cell state basic version is greater than 1 and revision version is not 0, illustrate that cell is modified, needs to revise non-mode database corresponding data;
Step (4): the schema modification SQL statement monitored according to daily record audiomonitor, mode regulator, by source relational data base schema modifications target non-mode database schema, goes to step (1).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210536616.1A CN102999629B (en) | 2012-12-12 | 2012-12-12 | A kind of relational database is to the asynchronous converting system of non-mode database and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210536616.1A CN102999629B (en) | 2012-12-12 | 2012-12-12 | A kind of relational database is to the asynchronous converting system of non-mode database and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102999629A CN102999629A (en) | 2013-03-27 |
| CN102999629B true CN102999629B (en) | 2016-01-13 |
Family
ID=47928196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210536616.1A Active CN102999629B (en) | 2012-12-12 | 2012-12-12 | A kind of relational database is to the asynchronous converting system of non-mode database and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102999629B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103955538B (en) * | 2014-05-19 | 2017-05-24 | 上海携程商务有限公司 | HBase data persistence and query methods and HBase system |
| CN104572505B (en) * | 2015-02-03 | 2017-05-17 | 济南大学 | System and method for ensuring eventual consistency of mass data caches |
| CN104598396B (en) * | 2015-02-03 | 2017-06-16 | 济南大学 | A kind of system and method for ensureing mass data caching real-time and uniformity |
| US10740318B2 (en) * | 2017-10-26 | 2020-08-11 | Sap Se | Key pattern management in multi-tenancy database systems |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7702698B1 (en) * | 2005-03-01 | 2010-04-20 | Yahoo! Inc. | Database replication across different database platforms |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
-
2012
- 2012-12-12 CN CN201210536616.1A patent/CN102999629B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7702698B1 (en) * | 2005-03-01 | 2010-04-20 | Yahoo! Inc. | Database replication across different database platforms |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
Non-Patent Citations (1)
| Title |
|---|
| NoSQL的分布式存储与扩展解决方法;姚林等;《计算机工程》;20120331;第38卷(第6期);第40-42页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102999629A (en) | 2013-03-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105183860B (en) | Method of data synchronization and system | |
| EP3365805B1 (en) | Ability to group multiple container databases as a single container database cluster | |
| CN102999629B (en) | A kind of relational database is to the asynchronous converting system of non-mode database and method | |
| Bolles et al. | Streaming SPARQL-extending SPARQL to process data streams | |
| CN103699638B (en) | Method for realizing cross-database type synchronous data based on configuration parameters | |
| US20180060389A1 (en) | Query optimization over distributed heterogeneous execution engines | |
| US9152665B2 (en) | Labeling versioned hierarchical data | |
| CN107038162A (en) | Real time data querying method and system based on database journal | |
| WO2008140937A3 (en) | Query handling in databases with replicated data | |
| CN103729447A (en) | Method for fast searching database | |
| WO2014035879A3 (en) | Operating a distributed database with foreign tables | |
| CN104598531A (en) | Incremental data migration method among heterogeneous relational databases based on trigger | |
| WO2012059816A3 (en) | Method and apparatus for handling digital objects in a communication network | |
| CN105335479A (en) | Text data statistics realization method based on SQL | |
| CN104484472A (en) | Database cluster for mixing various heterogeneous data sources and implementation method | |
| CN102708142A (en) | Method and system for acquiring background data through structured query language (SQL) custom-made template | |
| CN104021200A (en) | Data synchronizing method and device of database | |
| CN103412869A (en) | Method for batch pushing of server data updating | |
| CN107729410A (en) | A kind of database engine suitable for power equipment big data | |
| Yang | Data synchronization for integration systems based on trigger | |
| CN105354631A (en) | Optimization method for information assets device management of electrical power enterprise | |
| CN111695001A (en) | Mixed data management system in big data scene | |
| Choi et al. | Generating owl ontology from relational database | |
| CN202795370U (en) | Massive data processing system on basis of production real-time property | |
| CN105447057A (en) | Model transforming method and system based on business logic model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200701 Address after: Room 212, block a, 253 nanxinzhuang West Road, Shizhong District, Jinan City, Shandong Province Patentee after: Shandong Sizheng Information Technology Co., Ltd Address before: Ji'nan City Central Jiwei Road No. 106 of 250022 cities in Shandong Province Patentee before: University of Jinan |
|
| TR01 | Transfer of patent right |