CN105426396A - Routing algorithm based database sharding method, system and middleware system - Google Patents
Routing algorithm based database sharding method, system and middleware system Download PDFInfo
- Publication number
- CN105426396A CN105426396A CN201510713980.4A CN201510713980A CN105426396A CN 105426396 A CN105426396 A CN 105426396A CN 201510713980 A CN201510713980 A CN 201510713980A CN 105426396 A CN105426396 A CN 105426396A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- middleware
- title
- routing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention discloses a routing algorithm based database sharding method, system and middleware system. The method comprises: a database middleware receiving an access request of an application layer of a database, wherein the database middleware performs sharding to written data, and the written data is allocated to a database cluster according to a data routing; and the middleware reading the sharding data. The middleware system comprises: a database application connection module, which is used for receiving an instruction of the application layer and transferring data; a route processing module, which is used for sharding data and allocating a storage location by using the data routing; and a database cluster connection module, which is used for receiving the allocated storage location information and storing the sharding data in the corresponding database. According to the data routing based sharding method provided by the present invention, sharding is carried out on a large amount of data, and based on a sharding type and a data routing parameter calculation of the sharding type, target sharding data can be quickly accessed; and by using a reasonable sharding method, and a direct and efficient data lookup mechanism, processing ability and speed to a rapid expanded database are improved.
Description
Technical field
The present invention relates to a kind of database sharding method based on routing algorithm, system and middleware system, belong to database technical field.
Background technology
In information-technology age, data sharply expand, and for shopping website, such as user reaches after 1,000,000 grades, and in its station of bringing, the data relevant to individual subscriber such as letter, system message all will reach more than millions, even reach hundred million DBMSs.So start to adopt principal and subordinate (Master/Salve) read and write abruption technology, the pressure of distributing server.But the various aspects of performance expense between master-slave synchronisation is also very large.
In order to solve this difficult problem, data base administration person can carry out the vertical partitioning (point storehouse) of database on the basis of single Master/Salve pattern.So-called vertical partitioning refers to and according to the difference of business self, the traffic table of script redundancy in a database can be broken, data be stored in different databases respectively; Even if but be provided with index, still the fact that is excessive because of data volume thus the database performance caused decline cannot be covered, therefore this time, data base administration person just carried out horizontal partitioning (submeter) to database, so-called horizontal partitioning refers to and a traffic table is split into multiple sublist, is associated together between sublist by certain contract.
Even if adopt the method for point storehouse submeter to process database, along with the raising of Data Update speed and Data Update amount are that geometric sense increases, the CRUD operation that script database can complete, after point storehouse table mode, seem complicated and trouble, be reacted directly into data CRUD efficiency reduce above, wherein CRUD be increase (Create), read (Retrieve), upgrade (Update) and deletion (Delete) several word initial write a Chinese character in simplified form.
Summary of the invention
For prior art Problems existing, propose technical scheme of the present invention.
One aspect of the present invention provides a kind of database sharding method based on routing algorithm, comprises the following steps:
A, database middleware receive database application layer request of access, and described request of access comprises reading and writing data;
B, described database middleware to the data fragmentation of write, and according to data route assignment to data-base cluster;
C, described database middleware are according to the fragment data of described data route reading database cluster.
Before preferably database middleware receives database application layer request of access, also load default configuration file, configuration file comprises database application layer and middleware bind command table, data middleware and data-base cluster mapping table, data routing configuration rule.
Preferably database middleware is before the data fragmentation to write, and also comprise the type of setting burst, wherein, the type of burst comprises: a storehouse a slice and Ku Nei burst.
Preferably described database middleware is to after the data fragmentation of write, and data route calculates according to the title type of the type of burst and burst and distributes title to fragment data, and the fragment data being assigned with title is assigned to data-base cluster.
Preferably described burst title type comprises: title in overall title and cluster;
Wherein overall title refers to that the sheet of all fragment datas is called unique title;
In cluster, title refers to that the title of the fragment data of each database in data-base cluster is unique in this database.
Preferably described data route is according to the fragment data of data routing configuration rule reading database cluster, by fragment data set, and the data after set is exported.
Preferably resolve the SQL statement that described burst is corresponding, and the SQL statement after resolving is analyzed, to determine the data routing parameter of the fragment data being assigned with title;
The described routing parameter being assigned with the fragment data of title is sent to data route, performs routing procedure, to obtain the data meeting visiting demand according to described data routing configuration rule.
Another aspect of the present invention proposes a kind of middleware system based on data route, comprising:
Database application layer link block, for receiving instruction, the transmission data of application layer;
Route processing module, for by data route by data fragmentation and memory allocated position;
Data-base cluster link block, for receiving the stored position information of middleware routing layer distribution and fragment data being stored in correspondence database.
Described a kind of database separating system based on routing algorithm, comprising:
Database application module, for receive application software input data, to application software export data;
Database middleware module, receive the instruction of application layer, transmit data, by data route by data fragmentation and memory allocated position;
Data-base cluster module, for storing data.
Beneficial effect of the present invention is: based on the sharding method of data route, carries out burst to a large amount of data, and the data routing parameter based on the title type of burst type and burst calculates, can fast access target fragment data.By rational sharding method and direct effective data search mechanism, improve the processing power to the database of rapid expansion and speed.
Accompanying drawing explanation
In order to more clearly understand above-mentioned purpose of the present invention, feature and advantage, below in conjunction with the drawings and specific embodiments, the present invention is further described in detail, wherein:
The schematic diagram of fragmentation schema in the storehouse that Figure 1 shows that overall title according to a first embodiment of the present invention;
The schematic diagram of fragmentation schema in the storehouse that Figure 2 shows that title in cluster according to a second embodiment of the present invention;
Figure 3 shows that the schematic diagram of storehouse one tablet mode of overall title according to a third embodiment of the present invention;
Figure 4 shows that the schematic diagram of storehouse one tablet mode of title in cluster according to a fourth embodiment of the present invention;
Figure 5 shows that the Organization Chart of middleware system according to a fifth embodiment of the present invention.
Embodiment
According to the first embodiment of the present invention, for being carried out the method for burst by database middleware, comprise the following steps:
A, database middleware receive database application layer request of access, and described request of access comprises the requirements such as data reading, write, amendment, deletion.
B, database middleware are according to database application layer request of access, burst process is carried out to the data of write, some are divided into by data, simultaneously according to the type of burst and the title type of burst, the positional information corresponding by data route assignment one, and by fragment data distribution and transmission in data-base cluster, store.
C, database middleware are according to database application layer request of access, follow the title information according to the data required by database application layer, find data by data route, and fragment data is read from data-base cluster, and merging is transferred to database application layer in order.
Before wherein database middleware receives database application layer request of access, also load default configuration file, configuration file comprises database application layer and middleware bind command table, data middleware and data-base cluster mapping table, data routing configuration rule.
Database middleware is to after the data fragmentation of write, and data route calculates according to the title type of the type of burst and burst and distributes title to fragment data, and the fragment data being assigned with title is assigned to data-base cluster.
Wherein the type of burst comprises: a storehouse a slice and Ku Nei burst;
Burst title type comprises: title in overall title and cluster;
Wherein overall title refers to that the sheet of all fragment datas is called unique title;
The title that cluster internal sheet is called the fragment data of each database in data-base cluster is unique in this database.
Such as shown in Fig. 1, fragmentation schema in storehouse, can adopt overall title also can adopt title in cluster.If overall title, so hypothesis dbSize is 32, tbSize is 1024, and so the sheet number in each storehouse is 1024/32=32, and title distributes according to 0000-1023, and this just means that title is that the overall situation is unique, does not allow to duplicate.
According to a second embodiment of the present invention, such as Fig. 2 is shown in storehouse and adopts title in cluster in fragmentation schema, so suppose that dbSize is 32, tbSize is 1024, so the sheet number in each storehouse is 1024/32=32, and title is distribute according to 0-31 in each storehouse, that is, the non-overall situation is unique, such benefit, relative to the easier dilatation of overall title, Data Migration is also more prone to a little comparatively speaking, and in cluster, title more convenient data base administration person can do follow-up dilatation operation really.
According to the third embodiment of the invention, such as, shown in Fig. 3, storehouse one tablet mode, can adopt overall title or adopt title in cluster.If overall title, so hypothesis dbSize is 1024, tbSize is 1024, and so the sheet number in each storehouse is 1024/1024=1, and title distributes according to 0000-1023, and this just means that title is that the overall situation is unique, does not allow to duplicate.Relative to burst in storehouse, the algorithm of a storehouse a slice is simpler and directly perceived, because a storehouse a slice is only correlated with storehouse, has nothing to do, calculated storehouse, just equaled indirectly to calculate sheet with sheet.
According to a fourth embodiment of the invention, such as, shown in Fig. 4, in storehouse one tablet mode, adopt title in cluster, so hypothesis dbSize is 1024, tbSize is 1024, and so the sheet number in each storehouse is 1024/1024=1.That is, have 1024 storehouses, namely have 1024 sheets, but title is that the overall situation is consistent, without any suffix, all titles of the overall situation are all called 1.
Data after set, according to the fragment data of data routing configuration rule reading database cluster, by fragment data set, and export by data route.
Wherein resolve SQL statement corresponding to burst, and the SQL statement after resolving is analyzed, to determine the data routing parameter of the fragment data being assigned with title, i.e. symbol, the transmission path of data, the transmission mode of data, the parameter such as Data Position and order of representative data.
The described routing parameter being assigned with the fragment data of title is sent to data route, performs routing procedure, to obtain the data meeting visiting demand according to described data routing configuration rule.
The rules such as wherein routing configuration is regular is which type of parameter which type of content corresponding, how which type of parameter should process.
According to a fifth embodiment of the invention, be middleware system, comprise with lower module:
Database application layer link block, by the dissimilar instruction morphing instruction for effectively passing to middleware of types of applications software, for receiving instruction, the transmission data of application layer;
Route processing module, receives the processing instruction of application layer, according to routing algorithm and the routing rule of setting, by data route by data fragmentation and memory allocated position;
Data-base cluster link block, by the unified instruction used in described middleware, changes into the instruction of each types of database applicable, receives the stored position information of middleware routing layer distribution and fragment data is stored in correspondence database.
According to a sixth embodiment of the invention, a kind of database separating system based on routing algorithm, comprising:
Database application module, be configured to make database middleware receive database application layer request of access, described request of access comprises reading and writing data.Exemplarily, provide data-interface, data cube computation command library, for receive application software input data, to application software export data;
Database middleware module, is configured to the data fragmentation making described database middleware to write, and according to data route assignment to data-base cluster.Exemplarily, receive the instruction of application module, transmit data, by data route according to the routing algorithm of setting and routing rule by data fragmentation and memory allocated position;
Data-base cluster module, is configured to the fragment data making described database middleware according to described data route reading database cluster.Exemplarily, receive the data of middleware transmission and store data according to routing rule.
The above, just preferred embodiment of the present invention, the present invention is not limited to above-mentioned embodiment, as long as it reaches technique effect of the present invention with identical means, all should belong to protection scope of the present invention.In protection scope of the present invention, its technical scheme and/or embodiment can have various different modifications and variations.
Claims (9)
1., based on a database sharding method for routing algorithm, it is characterized in that, comprise the following steps:
A, database middleware receive database application layer request of access, and described request of access comprises reading and writing data;
B, described database middleware to the data fragmentation of write, and according to data route assignment to data-base cluster;
C, described database middleware are according to the fragment data of described data route reading database cluster.
2. a kind of database sharding method based on routing algorithm according to claim 1, is characterized in that, database middleware also comprises before receiving database application layer request of access:
Load the configuration file preset, wherein configuration file comprises:
Database application layer and middleware bind command table, data middleware and data-base cluster mapping table, data routing configuration rule.
3. a kind of database sharding method based on routing algorithm according to claim 1, it is characterized in that, database middleware, before the data fragmentation to write, also comprises the type of setting burst, wherein, the type of burst comprises: a storehouse a slice and Ku Nei burst.
4. a kind of database sharding method based on routing algorithm according to claim 1, it is characterized in that, described database middleware is to after the data fragmentation of write, data route calculates according to the title type of the type of burst and burst and distributes title to fragment data, and the fragment data being assigned with title is assigned to data-base cluster.
5. a kind of database sharding method based on routing algorithm according to claim 4, is characterized in that, described burst title type comprises: title in overall title and cluster;
Wherein overall title refers to that the sheet of all fragment datas is called unique title;
In cluster, title refers to that the title of the fragment data of each database in data-base cluster is unique in this database.
6. a kind of database sharding method based on routing algorithm according to claim 1, is characterized in that, the data after set, according to the fragment data of data routing configuration rule reading database cluster, by fragment data set, and export by described data route.
7. a kind of database sharding method based on routing algorithm according to claim 4 or 6, it is characterized in that, resolve the SQL statement that described burst is corresponding, and the SQL statement after resolving is analyzed, to determine the data routing parameter of the fragment data being assigned with title;
The described routing parameter being assigned with the fragment data of title is sent to data route, performs routing procedure, to obtain the data meeting visiting demand according to described data routing configuration rule.
8. a middleware system, is characterized in that, comprising:
Database application layer link block, for receiving instruction, the transmission data of application layer;
Route processing module, for by data route by data fragmentation and memory allocated position;
Data-base cluster link block, for receiving the stored position information of middleware routing layer distribution and fragment data being stored in correspondence database.
9., based on a database separating system for routing algorithm, it is characterized in that, comprising:
Database application module, be configured to make database middleware receive database application layer request of access, described request of access comprises reading and writing data;
Database middleware module, is configured to the data fragmentation making described database middleware to write, and according to data route assignment to data-base cluster;
Data-base cluster module, is configured to the fragment data making described database middleware according to described data route reading database cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510713980.4A CN105426396A (en) | 2015-10-28 | 2015-10-28 | Routing algorithm based database sharding method, system and middleware system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510713980.4A CN105426396A (en) | 2015-10-28 | 2015-10-28 | Routing algorithm based database sharding method, system and middleware system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105426396A true CN105426396A (en) | 2016-03-23 |
Family
ID=55504608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510713980.4A Pending CN105426396A (en) | 2015-10-28 | 2015-10-28 | Routing algorithm based database sharding method, system and middleware system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105426396A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202441A (en) * | 2016-07-13 | 2016-12-07 | 腾讯科技（深圳）有限公司 | Data processing method based on relevant database, device and system |
| CN106599043A (en) * | 2016-11-09 | 2017-04-26 | 中国科学院计算技术研究所 | Middleware used for multilevel database and multilevel database system |
| CN107180081A (en) * | 2017-04-28 | 2017-09-19 | 广东亿迅科技有限公司 | The call method and device of massive relation type database |
| CN107247760A (en) * | 2017-05-31 | 2017-10-13 | 民政部国家减灾中心 | Multi-layer mitigation disaster relief approaches to IM and system based on tight coupling |
| CN108153838A (en) * | 2017-12-15 | 2018-06-12 | 济南中维世纪科技有限公司 | A kind of MySQL database middleware preprocess method |
| CN108874837A (en) * | 2017-05-16 | 2018-11-23 | 北京京东尚科信息技术有限公司 | Database divides library method, apparatus, middleware and storage medium and electronic equipment |
| CN109344161A (en) * | 2018-12-04 | 2019-02-15 | 大唐网络有限公司 | A kind of mass data storage means based on mongodb |
| CN109828987A (en) * | 2019-01-21 | 2019-05-31 | 深圳乐信软件技术有限公司 | A kind of millions method for computing data, device, electronic equipment and medium |
| CN110059102A (en) * | 2019-04-25 | 2019-07-26 | 四川师范大学 | The method for manipulating adaptation resolver based on isomorphism type distributed data base integration CRUD |
| CN110674152A (en) * | 2019-09-24 | 2020-01-10 | 京东数字科技控股有限公司 | Data synchronization method and device, storage medium and electronic equipment |
| CN110928665A (en) * | 2019-12-06 | 2020-03-27 | 深圳乐信软件技术有限公司 | Data processing method, device, storage medium and terminal |
| CN110955666A (en) * | 2019-12-17 | 2020-04-03 | 用友网络科技股份有限公司 | Data processing method and data processing device |
| CN111274028A (en) * | 2020-01-15 | 2020-06-12 | 北大方正集团有限公司 | Partition method based on database middleware, partition device and readable storage medium |
| CN113495921A (en) * | 2020-04-02 | 2021-10-12 | 北京京东振世信息技术有限公司 | Routing method and device of database cluster |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120116743A1 (en) * | 2010-11-08 | 2012-05-10 | International Business Machines Corporation | Optimizing storage cloud environments through adaptive statistical modeling |
| CN103810265A (en) * | 2014-01-27 | 2014-05-21 | 南京邮电大学 | Database optimization method based on WiMAX local routing |
-
2015
- 2015-10-28 CN CN201510713980.4A patent/CN105426396A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120116743A1 (en) * | 2010-11-08 | 2012-05-10 | International Business Machines Corporation | Optimizing storage cloud environments through adaptive statistical modeling |
| CN103810265A (en) * | 2014-01-27 | 2014-05-21 | 南京邮电大学 | Database optimization method based on WiMAX local routing |
Non-Patent Citations (1)
| Title |
|---|
| 《微型机与应用》 * |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202441A (en) * | 2016-07-13 | 2016-12-07 | 腾讯科技（深圳）有限公司 | Data processing method based on relevant database, device and system |
| CN106599043A (en) * | 2016-11-09 | 2017-04-26 | 中国科学院计算技术研究所 | Middleware used for multilevel database and multilevel database system |
| CN107180081A (en) * | 2017-04-28 | 2017-09-19 | 广东亿迅科技有限公司 | The call method and device of massive relation type database |
| CN108874837B (en) * | 2017-05-16 | 2021-09-03 | 北京京东尚科信息技术有限公司 | Database partitioning method and device, middleware, storage medium and electronic equipment |
| CN108874837A (en) * | 2017-05-16 | 2018-11-23 | 北京京东尚科信息技术有限公司 | Database divides library method, apparatus, middleware and storage medium and electronic equipment |
| CN107247760A (en) * | 2017-05-31 | 2017-10-13 | 民政部国家减灾中心 | Multi-layer mitigation disaster relief approaches to IM and system based on tight coupling |
| CN108153838B (en) * | 2017-12-15 | 2022-03-11 | 山东中维世纪科技股份有限公司 | MySQL database middleware preprocessing method |
| CN108153838A (en) * | 2017-12-15 | 2018-06-12 | 济南中维世纪科技有限公司 | A kind of MySQL database middleware preprocess method |
| CN109344161A (en) * | 2018-12-04 | 2019-02-15 | 大唐网络有限公司 | A kind of mass data storage means based on mongodb |
| CN109828987A (en) * | 2019-01-21 | 2019-05-31 | 深圳乐信软件技术有限公司 | A kind of millions method for computing data, device, electronic equipment and medium |
| CN110059102A (en) * | 2019-04-25 | 2019-07-26 | 四川师范大学 | The method for manipulating adaptation resolver based on isomorphism type distributed data base integration CRUD |
| CN110674152A (en) * | 2019-09-24 | 2020-01-10 | 京东数字科技控股有限公司 | Data synchronization method and device, storage medium and electronic equipment |
| CN110928665A (en) * | 2019-12-06 | 2020-03-27 | 深圳乐信软件技术有限公司 | Data processing method, device, storage medium and terminal |
| CN110928665B (en) * | 2019-12-06 | 2023-06-06 | 深圳乐信软件技术有限公司 | Data processing method, device, storage medium and terminal |
| CN110955666A (en) * | 2019-12-17 | 2020-04-03 | 用友网络科技股份有限公司 | Data processing method and data processing device |
| CN110955666B (en) * | 2019-12-17 | 2023-10-03 | 用友网络科技股份有限公司 | Data processing method and data processing device |
| CN111274028A (en) * | 2020-01-15 | 2020-06-12 | 北大方正集团有限公司 | Partition method based on database middleware, partition device and readable storage medium |
| CN111274028B (en) * | 2020-01-15 | 2023-09-05 | 新方正控股发展有限责任公司 | Partitioning method, partitioning device and readable storage medium based on database middleware |
| CN113495921A (en) * | 2020-04-02 | 2021-10-12 | 北京京东振世信息技术有限公司 | Routing method and device of database cluster |
| CN113495921B (en) * | 2020-04-02 | 2023-09-26 | 北京京东振世信息技术有限公司 | Routing method and device for database cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105426396A (en) | Routing algorithm based database sharding method, system and middleware system | |
| CN107423422B (en) | Spatial data distributed storage and search method and system based on grid | |
| CN102663117B (en) | OLAP (On Line Analytical Processing) inquiry processing method facing database and Hadoop mixing platform | |
| CN102122285B (en) | Data cache system and data inquiry method | |
| CN102968503B (en) | The data processing method of Database Systems and Database Systems | |
| US9547706B2 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| US9501506B1 (en) | Indexing system | |
| CN103067461B (en) | A kind of metadata management system of file and metadata management method | |
| CN103067433B (en) | A kind of data migration method of distributed memory system, equipment and system | |
| CN112363979B (en) | Distributed index method and system based on graph database | |
| CN102509012A (en) | Method for mapping contents of electronic medical record into electronic medical record standard database | |
| CN104395904A (en) | Efficient data object storage and retrieval | |
| US9405643B2 (en) | Multi-level lookup architecture to facilitate failure recovery | |
| CN105612518A (en) | Methods and systems for autonomous memory searching | |
| CN102169507A (en) | Distributed real-time search engine | |
| CN110633378A (en) | Graph database construction method supporting super-large scale relational network | |
| CN103258036A (en) | Distributed real-time search engine based on p2p | |
| JP2013196565A (en) | Database processing method, and database processor | |
| CN103440301A (en) | Data multi-duplicate hybrid storage method and system | |
| CN105677761A (en) | Data sharding method and system | |
| CN102411634A (en) | Data storage method for improving instantaneity of embedded type database | |
| US20150169623A1 (en) | Distributed File System, File Access Method and Client Device | |
| CN104410531B (en) | The system architecture method of redundancy | |
| CN104112011A (en) | Method and device for extracting mass data | |
| CN105677904A (en) | Distributed file system based small file storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160323 |