CN107229688A - A kind of database level point storehouse point table method and system, server - Google Patents
A kind of database level point storehouse point table method and system, server Download PDFInfo
- Publication number
- CN107229688A CN107229688A CN201710334450.8A CN201710334450A CN107229688A CN 107229688 A CN107229688 A CN 107229688A CN 201710334450 A CN201710334450 A CN 201710334450A CN 107229688 A CN107229688 A CN 107229688A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- cluster
- data record
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/213—Schema design and management with details for schema evolution support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of database level point storehouse point table method and system, server, comprises the following steps：Set the number of cluster, the number for the form that the database number that a cluster is included and a data place are included；Routing rule is split based on presetting database, data record is distributed in each database included to each cluster；Routing rule is split based on preset table, the data record in each database is distributed in each form included to the data place.The database level point storehouse point line number of table method and system, server based on single table of the present invention, database is horizontally divided into some points of storehouses, each point storehouse is divided into some points of tables again, so as to reduce the record quantity of single table, improve the response time required for inquiry, and be easy to extension.
Description
Technical field
The present invention relates to the technical field of database, more particularly to a kind of database level point storehouse divides table method and is
System, server.
Background technology
Database (Database) is to come tissue, storage and the foundation of management data according to data structure to store in computer
Warehouse in equipment.Generally, database is the general data processing system of a unit or an application field, what it was stored
It is to belong to enterprise and operating divisions, the set of the relevant data of organization and individual.Data in database are gone out from the overall point of view
What hair was set up, carry out tissue, description by certain data model and store；Its structure based on the associate naturally between data so that
All necessary access paths can be provided, and data are no longer directed to a certain application, but towards full tissue, with overall structure
Change feature.
As business is continued to develop, the business datum of application system is also with accumulating over a long period.Corresponding performance indications are continuous
Lifting, increasingly finely, such as credit examination ＆ verification automation, the pick of the full dimension portrait information of user take.Liter in these business demands
Level, the requirement to the response time is increasingly faster, causes the requirement more and more higher to systematic function.
Former database is divided into some divided data storehouses by vertical point storehouse according to business scope, so that between solving operation system
Coupling, also can improving performance to a certain degree, be conducive to system maintenance.Vertically a point table is that usually said big table tears small table open, is torn open
Divide is based on the row progress in relevant database.When the field in some table is relatively more, an expansion table can be newly set up,
The field being not frequently used or length is larger is split away and is put into expansion table.Therefore, in the case where field is especially more, hang down
Directly a point table is easy to develop and safeguard really, in some sense may surface is " cross-page " the problem of access.
As shown in figure 1, the database schema after the table of vertical point of storehouse of the prior art point generally uses master-slave
Pattern, can only be extended to the read operation of database, and database write operation is still concentrated on master, single
The upper carry slave of master quantity, is also limited by master performances.
The mode of vertical point storehouse point table alleviates the pressure of original single cluster, but can not solve the second and the online query field such as kill
Scape.For example need to judge a user, it is interior during some activity, only allow next list.Such scene is looked into order table
Ask, the requirement of response time is just very harsh.For high table as order, performance will turn into inquiry bottleneck.Relationship type
Database is retrieved performance in the case of more than certain data volume and can drastically declined.When in face of internet mass data cases,
All data are all stored in a table, it is clear that can readily exceed the data volume threshold values that database table can be born.
For the Internet, applications, the record line number of database list table, after some time it is possible to reach ten million rank, even with hundred million grades
Not, thus can to real time business inquiry cause to have a strong impact on.Therefore, point storehouse point table being carried out to database must be carried out.For
Access for the extremely huge single table of frequent, data volume, primary is exactly the record strip number for reducing single table, required to reduce inquiry
Response time, improve data throughput.And vertically point storehouse point table method can not solve this problem to existing database.
The content of the invention
The shortcoming of prior art, divides table it is an object of the invention to provide a kind of database level point storehouse in view of the above
Method and system, server, based on the line number of single table, some points of storehouses are horizontally divided into by database, and each point storehouse is divided into again
Some points of tables, so as to reduce the record quantity of single table, are improved the response time required for inquiry, and be easy to extension.
In order to achieve the above objects and other related objects, the present invention provides a kind of database level point storehouse point table method, bag
Include following steps：The table that the database number and a data place that the number of setting cluster, a cluster are included are included
The number of lattice；Routing rule is split based on presetting database, data record is distributed to each data included to each cluster
In storehouse；Routing rule is split based on preset table, distributes what is included to the data place by the data record in each database
In each form.
In one embodiment of the invention, the form that the database number that the cluster is included is included with the data place
Number value it is identical.
In one embodiment of the invention, routing rule is split based on presetting database, data record is distributed to each collection
Comprise the following steps in each database that group is included：
Database linear expansion multiplication factor Factor=Round (userId/scope) is calculated, wherein, Round is represented
Downward rounding operation, useId represents uniquely corresponding and with the user of digital record incremental digital record one by one with data record
Numbering, scope represents the data recording capability number of a cluster；
Hash cutting factor R emainder=userId mod scope are calculated, wherein, mod represents to take the remainder operation；
Remainder mod n+Factor x n are calculated, database residing for data record are obtained in its residing cluster
Numbering, the data record is distributed into the corresponding database of the numbering, wherein, the database that n is included by a cluster
Number.
In one embodiment of the invention, routing rule is split based on preset table, by the data record in each database
Distribute and comprise the following steps in each form included to the data place：
The data record obtained in database follows a certain regular number value；
The number for the form that the number value is included to data place carries out remainder operation, obtains form residing for data record
Numbering in its residing database, the data record is distributed into the corresponding form of the numbering.
In one embodiment of the invention, the described number value uses the useId values or orderId values of data record；Wherein,
UseId represents uniquely corresponding and with the Customs Assigned Number of digital record incremental digital record one by one, orderId tables with data record
Show uniquely corresponding and with the O/No. of digital record incremental digital record one by one with data record.
Meanwhile, the present invention also provides a kind of database level point storehouse point table system, including setup module, database split mould
Block and form split module；
The setup module is for setting the number of cluster, the database number and a data that a cluster is included
The number for the form that place is included；
The database, which splits module, to be used to split routing rule based on presetting database, and data record is distributed to each
In each database that cluster is included；
The form, which splits module, to be used to split routing rule based on preset table, by the data record in each database
Distribute in each form included to the data place.
In one embodiment of the invention, the setup module sets database number and the number that the cluster is included
The value of the number of the form included according to place is identical.
In one embodiment of the invention, the database splits module and performs following operate：：
Database linear expansion multiplication factor Factor=Round (userId/scope) is calculated, wherein, Round is represented
Downward rounding operation, useId represents uniquely corresponding and with the user of digital record incremental digital record one by one with data record
Numbering, scope represents the data recording capability number of a cluster；
Hash cutting factor R emainder=userId mod scope are calculated, wherein, mod represents to take the remainder operation；
Remainder mod n+Factor x n are calculated, database residing for data record are obtained in its residing cluster
Numbering, the data record is distributed into the corresponding database of the numbering, wherein, the database that n is included by a cluster
Number.
In one embodiment of the invention, the form splits module and performs following operate：
The data record obtained in database follows a certain regular number value；
The number for the form that the number value is included to data place carries out remainder operation, obtains form residing for data record
Numbering in its residing database, the data record is distributed into the corresponding form of the numbering.
In addition, the present invention also provides a kind of server, including any of the above-described described database level point storehouse point table system.
As described above, database level point storehouse point table method and system, the server of the present invention, with following beneficial effect
Really：
(1) line number based on single table, some points of storehouses are horizontally divided into by database, and each point storehouse is divided into some points again
Table, so as to reduce the record quantity of single table, improves the response time required for inquiry；
(2) deployment is convenient, it is low, invasive without code to develop the cost such as integrated；
(3) database linear expansion is supported, no data migration pain, dilatation cost is low；
(4) can just it implement according to the size active dilatation of the amount of storage of setting, rather than when bottleneck occurs in database,
It ensure that the security reliability of database；
(5) a data need to only be migrated, you can realize the renewal of the database after former database to level point storehouse point table.
Brief description of the drawings
Fig. 1 is shown as the schematic diagram in the vertical point storehouse of database in the prior art；
Fig. 2 is shown as the flow chart of the database level point storehouse point table method of the present invention；
Fig. 3 is shown as the structural representation of the database level point storehouse point table system of the present invention；
Fig. 4 is shown as the structural representation of the server of the present invention.
Component label instructions
1 setup module
2 databases split module
3 forms split module
Embodiment
Illustrate embodiments of the present invention below by way of specific instantiation, those skilled in the art can be by this specification
Disclosed content understands other advantages and effect of the present invention easily.The present invention can also pass through specific realities different in addition
The mode of applying is embodied or practiced, the various details in this specification can also based on different viewpoints with application, without departing from
Various modifications or alterations are carried out under the spirit of the present invention.
It should be noted that the diagram provided in the present embodiment only illustrates the basic conception of the present invention in a schematic way,
Then only display is painted with relevant component in the present invention rather than according to component count, shape and the size during actual implementation in schema
System, it is actual when implementing, and kenel, quantity and the ratio of each component can be a kind of random change, and its assembly layout kenel also may be used
Can be increasingly complex.
The database level point storehouse point line number of table method and system, server based on single table of the present invention, by database water
Divide equally and be segmented into some points of storehouses, each point storehouse is divided into some points of tables again, so as to reduce the record quantity of single table, improve inquiry required
The response time wanted, and database linear expansion is supported, no data migration pain, dilatation cost is low.
Reference picture 2, database level point storehouse point table method of the invention comprises the following steps：
The database number and a data place that step S1, the number for setting cluster, a cluster are included are included
Form number.
Specifically, according to the bar number of the actual data record produced, the number that the number of cluster, a cluster are included is determined
The number of the form included according to storehouse number and a data place.In actual use, can be according to actual data record
Bar number carries out the linear expansion of database.It should be noted that the principle of linear expansion is the premise for ensureing single table response performance
Under, while taking into account benefit.
So that order data is recorded as an example, it is assumed that current order table includes 7,000,000 records.As shown in table 1, torn open using 4x4
During the cluster divided, single table storage is recorded as 6,250,000.
Corresponding single table storing capacity under table 1, different storehouse dimensions and different table dimensions
| Total capacity (hundred million) | Storehouse dimension | Table dimension | Table quantity in cluster | Single table storage record number (ten thousand) |
| 100,000,000 | 4 | 4 | 16 | 6,250,000 |
| 1,000,000,000 | 8 | 8 | 64 | 15,625,000 |
| 10,000,000,000 | 16 | 16 | 256 | 39,062,500 |
| 100,000,000,000 | 32 | 32 | 1024 | 97,656,250 |
(current monthly average 700,000) is calculated by monthly 2,000,000 orders, 10 years orders can reach 2.47 hundred million records.Hash
8 table storages of 8x, the record number of every table storage is recorded less than then the performance of single table should be in young shape for 3,860,000
State, can meet the demand of the applications such as actual queries.For 2.47 hundred million records after 10 years, as shown in table 2, dilatation in the middle of 10 years
Secondary, i.e., from 4x4 dilatations to 8x8, then dilatation is to 16x16.
Corresponding single table storing capacity under table 2, different storehouse dimensions and different table dimensions
| Total capacity (hundred million) | Storehouse dimension | Table dimension | Table quantity in cluster | Single table storage record number (ten thousand) |
| 500,000,000 | 4 | 4 | 16 | 31,250,000 |
| 500,000,000 | 8 | 8 | 64 | 7,812,500 |
| 500,000,000 | 16 | 16 | 256 | 1,953,125 |
| 500,000,000 | 32 | 32 | 1024 | 488,281 |
Preferably, the database number that cluster is included is identical with the value of the number for the form that data place is included.
Step S2, based on presetting database split routing rule, by data record distribute to each cluster included it is each
In individual database.
Specifically, based on presetting database split routing rule, by data record distribute to each cluster included it is each
Comprise the following steps in individual database：
21) database linear expansion multiplication factor Factor=Round (userId/scope) is calculated, wherein, Round tables
Show downward rounding operation, useId represents uniquely corresponding and with the use of digital record incremental digital record one by one with data record
Family is numbered, and scope represents the data recording capability number of a cluster.
Wherein, i.e. cluster residing for database linear expansion multiplication factor Factor data records numbering.Cluster
Numbering is since 0, and incremented by successively 1.
22) hash cutting factor R emainder=userId mod scope are calculated, wherein, mod represents to take the remainder behaviour
Make.
Wherein, Remainder as useId hash values, for follow-up hash cuttings.
23) Remainder mod n+Factor x n are calculated, database residing for data record are obtained in its residing cluster
In numbering so that the data record is distributed into the corresponding database of the numbering, wherein, the data that n is included by a cluster
Storehouse number.
For example, it is desired to when a cluster is cut into 4 databases, just with hash value of 4 this numeral to useId
Remainder carries out taking the remainder computing, that is, Remainder mod 4.Like this computing just has four kinds of possibility every time：Knot
When fruit is 0, correspondence DB1；As a result correspondence DB2 when being 1；As a result correspondence DB3 when being 2；As a result be 3 when pair
Answer DB4.So just it is highly uniform by data distribution into 4 databases, be conducive to follow-up data retrieval.
It should be noted that database accession number linear expansion in each cluster, with continuity.
Therefore, by above-mentioned steps, the linear expansion of database, the linear dilatation of cluster are realized.Simultaneously as in collection
Hash segmentation algorithms are used in group so that distribution is highly uniform to recording for each database.
Step S3, based on preset table split routing rule, the data record in each database is distributed to the data
In each form that place is included.
Specifically, routing rule is split based on preset table, the data record in each database is distributed to the data
Comprise the following steps in each form that place is included：
31) data record obtained in database follows a certain regular number value.
Specifically, the number value can be the orderId values of above-mentioned useId values or data record.Wherein,
OrderId represents uniquely corresponding and with the O/No. of digital record incremental digital record one by one with data record.
It should be noted that form splits routing rule and database splits routing rule and uncorrelated, therefore form is split
Route can be using parameters such as orderId.Because, in actual applications, order inquiries are common scenes.Press
The scene frequency of orderId inquiries is also very high.It is of course also possible to continue to do Hash cuttings with userId values.
32) number for the form for including the number value to data place carries out remainder operation, obtains residing for data record
Numbering of the form in its residing database, the data record is distributed into the corresponding form of the numbering.
In form splits routing rule, still using hash cuttings, routing rule is split with database so as to ensure that
Uniformity.This causes after the horizontal cutting of data record, it is easy to carry out horizontal extension, is looked into while can also take into account according to orderId
The performance of inquiry, therefore greatly improve Consumer's Experience.
The database level point storehouse point table method of the present invention is expanded on further below by specific embodiment.
Setting includes two clusters, and each cluster includes four databases, and each database includes four forms.Each collection
The capacity for the data record that group is included is 10000.In this embodiment, specific database level point storehouse point table method such as table
Shown in 3.
The specific embodiment of table 3, database level point storehouse point table method
When useId is respectively 9900,9901,19900 and 19901, the corresponding data record allocation result such as institute of table 4
Show.
Table 4, data record allocation result
Therefore, by the present invention database level point storehouse divide table method each data record can accordingly be distributed to
Each specific form, realizes the level point storehouse point table to the data record of raw data base, and database linear expansion, nothing
Data Migration pain, dilatation cost is low.
In addition, for existing database, it is necessary to be stored data record to level point storehouse by way of Data Migration
The database divided after table.The Data Migration need to only be carried out once, specifically include following steps：
Stage one
The coding rules such as order number are determined, when the old and new's database is inserted, the order number generation rule of the old and new's database are judged
It is then whether consistent, can be using the time as section if inconsistent cannot be as Data Migration section.
In original service code, the data source write-in of new burst is added, the double landings of data are realized.Inquiry is read holding and walked
Old database source.Affairs successfully judge to be defined by old database.
Migrating data section is determined, double landings in principle and the data variance that this time window causes is migrated get over
It is small better.Double drop on-line time point 2016.12.31 23.59.59 are such as taken, as the section of migration history data, this
Data before individual time point are, it is necessary to migrate.
Stage two
Data Migration terminates, and new order (also error compensation), needs to check whether and deposit before insertion in compensation time window
.
Data consistency in operation a period of time, regular check the old and new's data source, finds reason, is handled in time,
And compensation.
A period of time is lasted, determines that data source change is read in switching after the indifference opposite sex, is altered to fragment data source.Affairs success
It is defined by new database.
Stage three
Switch the application relied on old data source, close old data source and write, become singly to fall, only fall the new database of burst.
Old database can cancel.
It should be noted that the need of work for carrying out Data Migration meets claimed below：
(1) need to dock the new database of old database and burst；
(2) it can support by input SQL statement migration, that is, support temporally or ID segmentations are migrated；
(3) when importing error, orderId is recorded, to compensate；
(4) compensation migration, it is necessary to do Exists inspections before insertion.Just start high-volume to migrate, it is not necessary to be Exists
Check, otherwise can influence very much efficiency.Judgement is determined whether to by increasing configuration parameter.
Reference picture 3, database level point storehouse point table system of the invention includes setup module 1, the database being sequentially connected
Split module 2 and form splits module 3.
Setup module 1 is used to set the number of cluster, the database number that a cluster is included and a database
Comprising form number.
Specifically, according to the bar number of the actual data record produced, the number that the number of cluster, a cluster are included is determined
The number of the form included according to storehouse number and a data place.In actual use, can be according to actual data record
Bar number carries out the linear expansion of database.It should be noted that the principle of linear expansion is the premise for ensureing single table response performance
Under, while taking into account benefit.
So that order data is recorded as an example, it is assumed that current order table includes 7,000,000 records.As shown in table 1, torn open using 4x4
During the cluster divided, single table storage is recorded as 6,250,000.
Corresponding single table storing capacity under table 1, different storehouse dimensions and different table dimensions
| Total capacity (hundred million) | Storehouse dimension | Table dimension | Table quantity in cluster | Single table storage record number (ten thousand) |
| 100,000,000 | 4 | 4 | 16 | 6,250,000 |
| 1,000,000,000 | 8 | 8 | 64 | 15,625,000 |
| 10,000,000,000 | 16 | 16 | 256 | 39,062,500 |
| 100,000,000,000 | 32 | 32 | 1024 | 97,656,250 |
(current monthly average 700,000) is calculated by monthly 2,000,000 orders, 10 years orders can reach 2.47 hundred million records.Hash
8 table storages of 8x, the record number of every table storage is recorded less than then the performance of single table should be in young shape for 3,860,000
State, can meet the demand of the applications such as actual queries.For 2.47 hundred million records after 10 years, as shown in table 2, dilatation in the middle of 10 years
Secondary, i.e., from 4x4 dilatations to 8x8, then dilatation is to 16x16.
Corresponding single table storing capacity under table 2, different storehouse dimensions and different table dimensions
| Total capacity (hundred million) | Storehouse dimension | Table dimension | Table quantity in cluster | Single table storage record number (ten thousand) |
| 500,000,000 | 4 | 4 | 16 | 31,250,000 |
| 500,000,000 | 8 | 8 | 64 | 7,812,500 |
| 500,000,000 | 16 | 16 | 256 | 1,953,125 |
| 500,000,000 | 32 | 32 | 1024 | 488,281 |
Preferably, the database number that cluster is included is identical with the value of the number for the form that data place is included.
Database, which splits module 2, to be used to split routing rule based on presetting database, and data record is distributed to each collection
In each database that group is included.
Specifically, database splits module 2 and performs following operation：
21) database linear expansion multiplication factor Factor=Round (userId/scope) is calculated, wherein, Round tables
Show downward rounding operation, useId represents uniquely corresponding and with the use of digital record incremental digital record one by one with data record
Family is numbered, and scope represents the data recording capability number of a cluster.
Wherein, i.e. cluster residing for database linear expansion multiplication factor Factor data records numbering.Cluster
Numbering is since 0, and incremented by successively 1.
22) hash cutting factor R emainder=userId mod scope are calculated, wherein, mod represents to take the remainder behaviour
Make.
Wherein, Remainder as useId hash values, for follow-up hash cuttings.
23) Remainder mod n+Factor x n are calculated, database residing for data record are obtained in its residing cluster
In numbering so that the data record is distributed into the corresponding database of the numbering, wherein, the data that n is included by a cluster
Storehouse number.
For example, it is desired to when a cluster is cut into 4 databases, just with hash value of 4 this numeral to useId
Remainder carries out taking the remainder computing, that is, Remainder mod 4.Like this computing just has four kinds of possibility every time：Knot
When fruit is 0, correspondence DB1；As a result correspondence DB2 when being 1；As a result correspondence DB3 when being 2；As a result be 3 when pair
Answer DB4.So just it is highly uniform by data distribution into 4 databases, be conducive to follow-up data retrieval.
It should be noted that database accession number linear expansion in each cluster, with continuity.
Therefore, by above-mentioned steps, the linear expansion of database, the linear dilatation of cluster are realized.Simultaneously as in collection
Hash segmentation algorithms are used in group so that distribution is highly uniform to recording for each database.
Form, which splits module 3, to be used to split routing rule based on preset table, by the data record in each database point
It is assigned in each form that the data place is included.
Specifically, form splits module and performs following operate：
31) data record obtained in database follows a certain regular number value.
Specifically, the number value can be the orderId values of above-mentioned useId values or data record.Wherein,
OrderId represents uniquely corresponding and with the O/No. of digital record incremental digital record one by one with data record.
It should be noted that form splits routing rule and database splits routing rule and uncorrelated, therefore form is split
Route can be using parameters such as orderId.Because, in actual applications, order inquiries are common scenes.Press
The scene frequency of orderId inquiries is also very high.It is of course also possible to continue to do Hash cuttings with userId values.
32) number for the form for including the number value to data place carries out remainder operation, obtains residing for data record
Numbering of the form in its residing database, the data record is distributed into the corresponding form of the numbering.
In form splits routing rule, still using hash cuttings, routing rule is split with database so as to ensure that
Uniformity.This causes after the horizontal cutting of data record, it is easy to carry out horizontal extension, is looked into while can also take into account according to orderId
The performance of inquiry, therefore greatly improve Consumer's Experience.
As shown in figure 4, the present invention also provides a kind of server, including above-mentioned database level point storehouse point table system.
In summary, the database level point storehouse point line number of table method and system, server based on single table of the invention, will
Database is horizontally divided into some points of storehouses, and each point storehouse is divided into some points of tables again, so as to reduce the record quantity of single table, improves
Response time required for inquiry；Deployment is convenient, it is low, invasive without code to develop the cost such as integrated；Database is supported linearly to expand
Exhibition, no data migration pain, dilatation cost is low；Can be according to the size active dilatation of the amount of storage of setting, rather than in data
Just implementing during bottleneck occurs in storehouse, it is ensured that the security reliability of database；A data only need to be migrated, you can realize former database
The renewal of database after to level point storehouse point table.So, the present invention effectively overcomes various shortcoming of the prior art and had
High industrial utilization.
The above-described embodiments merely illustrate the principles and effects of the present invention, not for the limitation present invention.It is any ripe
Know the personage of this technology all can carry out modifications and changes under the spirit and scope without prejudice to the present invention to above-described embodiment.Cause
This, those of ordinary skill in the art is complete without departing from disclosed spirit and institute under technological thought such as
Into all equivalent modifications or change, should by the present invention claim be covered.
Claims (10)
1. a kind of database level point storehouse point table method, it is characterised in that：Comprise the following steps：
Set the number of cluster, for the form that the database number that a cluster is included and a data place are included
Number；
Routing rule is split based on presetting database, data record is distributed in each database included to each cluster；
Based on preset table split routing rule, by the data record in each database distribute to the data place include it is each
In individual form.
2. database level according to claim 1 point storehouse point table method, it is characterised in that：The number that the cluster is included
It is identical with the value of the number for the form that the data place is included according to storehouse number.
3. database level according to claim 1 point storehouse point table method, it is characterised in that：Split based on presetting database
Routing rule, data record is distributed and comprised the following steps in each database included to each cluster：
Database linear expansion multiplication factor Factor=Round (userId/scope) is calculated, wherein, Round represents downward
Rounding operation, useId represents uniquely corresponding and with the Customs Assigned Number of digital record incremental digital record one by one with data record,
Scope represents the data recording capability number of a cluster；
Hash cutting factor R emainder=userId mod scope are calculated, wherein, mod represents to take the remainder operation；
Remainder mod n+Factor x n are calculated, volume of the database residing for data record in its residing cluster is obtained
Number, the data record is distributed into the corresponding database of the numbering, wherein, the database that n is included by a cluster
Number.
4. database level according to claim 1 point storehouse point table method, it is characterised in that：Road is split based on preset table
By rule, the data record in each database is distributed and comprised the following steps in each form included to the data place：
The data record obtained in database follows a certain regular number value；
The number for the form that the number value is included to data place carries out remainder operation, obtains form residing for data record at it
Numbering in residing database, the data record is distributed into the corresponding form of the numbering.
5. database level according to claim 4 point storehouse point table method, it is characterised in that：The described number value uses number
According to the useId values or orderId values of record；Wherein, useId represents uniquely corresponding and passed one by one with digital record with data record
The Customs Assigned Number of the digital record of increasing, orderId represents uniquely corresponding and with digital record incremental number one by one with data record
The O/No. of word record.
6. a kind of database level point storehouse point table system, it is characterised in that：Module and form are split including setup module, database
Split module；
The setup module is for setting the number of cluster, the database number and a data place that a cluster is included
Comprising form number；
The database, which splits module, to be used to split routing rule based on presetting database, and data record is distributed to each cluster
Comprising each database in；
The form, which splits module, to be used to split routing rule based on preset table, and the data record in each database is distributed
In each form included to the data place.
7. database level according to claim 6 point storehouse point table system, it is characterised in that：The setup module sets institute
State the database number that cluster included identical with the value of the number for the form that the data place is included.
8. database level according to claim 6 point storehouse point table system, it is characterised in that：The database splits module
Perform following operate：：
Database linear expansion multiplication factor Factor=Round (userId/scope) is calculated, wherein, Round represents downward
Rounding operation, useId represents uniquely corresponding and with the Customs Assigned Number of digital record incremental digital record one by one with data record,
Scope represents the data recording capability number of a cluster；
Hash cutting factor R emainder=userId mod scope are calculated, wherein, mod represents to take the remainder operation；
Remainder mod n+Factor x n are calculated, volume of the database residing for data record in its residing cluster is obtained
Number, the data record is distributed into the corresponding database of the numbering, wherein, the database that n is included by a cluster
Number.
9. database level according to claim 6 point storehouse point table system, it is characterised in that：The form splits module and held
Row is following to be operated：
The data record obtained in database follows a certain regular number value；
The number for the form that the number value is included to data place carries out remainder operation, obtains form residing for data record at it
Numbering in residing database, the data record is distributed into the corresponding form of the numbering.
10. a kind of server, it is characterised in that：Including the database level point storehouse point table system described in one of claim 6-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710334450.8A CN107229688A (en) | 2017-05-12 | 2017-05-12 | A kind of database level point storehouse point table method and system, server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710334450.8A CN107229688A (en) | 2017-05-12 | 2017-05-12 | A kind of database level point storehouse point table method and system, server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107229688A true CN107229688A (en) | 2017-10-03 |
Family
ID=59933374
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710334450.8A Pending CN107229688A (en) | 2017-05-12 | 2017-05-12 | A kind of database level point storehouse point table method and system, server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107229688A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344161A (en) * | 2018-12-04 | 2019-02-15 | 大唐网络有限公司 | A kind of mass data storage means based on mongodb |
| CN109783577A (en) * | 2019-01-05 | 2019-05-21 | 咪付（广西）网络技术有限公司 | A kind of cloud database elastic telescopic method based on strategy |
| CN109918442A (en) * | 2019-01-24 | 2019-06-21 | 中国联合网络通信集团有限公司 | A kind of processing method and processing device of data |
| CN110134742A (en) * | 2019-04-25 | 2019-08-16 | 上海淇毓信息科技有限公司 | A kind of dynamic global level grouping method, device, system based on financial client |
| CN110222030A (en) * | 2019-05-13 | 2019-09-10 | 福建天泉教育科技有限公司 | The method of Database Dynamic dilatation, storage medium |
| CN111199386A (en) * | 2019-12-27 | 2020-05-26 | 天阳宏业科技股份有限公司 | Workflow engine and implementation method thereof |
| CN111753019A (en) * | 2019-03-26 | 2020-10-09 | 北京沃东天骏信息技术有限公司 | Data partitioning method and device applied to data warehouse |
| CN112184204A (en) * | 2020-09-30 | 2021-01-05 | 银盛支付服务股份有限公司 | Data operation method and system based on data routing rule and database and table division |
| CN117473071A (en) * | 2023-12-27 | 2024-01-30 | 珠海格力电器股份有限公司 | Data retrieval method, device, equipment and computer readable medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541997A (en) * | 2011-11-23 | 2012-07-04 | 苏州阔地网络科技有限公司 | Database execution method and system for automatic database partitioning |
| CN102930062A (en) * | 2012-11-30 | 2013-02-13 | 南京富士通南大软件技术有限公司 | Rapid horizontal extending method for databases |
| CN102999526A (en) * | 2011-09-16 | 2013-03-27 | 阿里巴巴集团控股有限公司 | Splitting and inquiring method and system of database relational table |
| US8429162B1 (en) * | 2011-09-27 | 2013-04-23 | Amazon Technologies, Inc. | Facilitating data redistribution in database sharding |
| CN105930387A (en) * | 2016-04-14 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Data operation system and method based on data routing and sharding |
| CN106528579A (en) * | 2015-09-15 | 2017-03-22 | 阿里巴巴集团控股有限公司 | Search method, device and system based on sharding structure databases |
-
2017
- 2017-05-12 CN CN201710334450.8A patent/CN107229688A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999526A (en) * | 2011-09-16 | 2013-03-27 | 阿里巴巴集团控股有限公司 | Splitting and inquiring method and system of database relational table |
| US8429162B1 (en) * | 2011-09-27 | 2013-04-23 | Amazon Technologies, Inc. | Facilitating data redistribution in database sharding |
| CN102541997A (en) * | 2011-11-23 | 2012-07-04 | 苏州阔地网络科技有限公司 | Database execution method and system for automatic database partitioning |
| CN102930062A (en) * | 2012-11-30 | 2013-02-13 | 南京富士通南大软件技术有限公司 | Rapid horizontal extending method for databases |
| CN106528579A (en) * | 2015-09-15 | 2017-03-22 | 阿里巴巴集团控股有限公司 | Search method, device and system based on sharding structure databases |
| CN105930387A (en) * | 2016-04-14 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Data operation system and method based on data routing and sharding |
Non-Patent Citations (2)
| Title |
|---|
| ZHENGDL126: "数据库水平切分的实现原理解析－－－分库，分表，主从，集群，负载均衡器", 《ITEYE》 * |
| 天梯梦: "大数据量的存储分表常见算法", 《ITEYE》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344161A (en) * | 2018-12-04 | 2019-02-15 | 大唐网络有限公司 | A kind of mass data storage means based on mongodb |
| CN109783577A (en) * | 2019-01-05 | 2019-05-21 | 咪付（广西）网络技术有限公司 | A kind of cloud database elastic telescopic method based on strategy |
| CN109783577B (en) * | 2019-01-05 | 2021-10-08 | 咪付（广西）网络技术有限公司 | Strategy-based cloud database elastic expansion method |
| CN109918442A (en) * | 2019-01-24 | 2019-06-21 | 中国联合网络通信集团有限公司 | A kind of processing method and processing device of data |
| CN111753019A (en) * | 2019-03-26 | 2020-10-09 | 北京沃东天骏信息技术有限公司 | Data partitioning method and device applied to data warehouse |
| CN110134742A (en) * | 2019-04-25 | 2019-08-16 | 上海淇毓信息科技有限公司 | A kind of dynamic global level grouping method, device, system based on financial client |
| CN110134742B (en) * | 2019-04-25 | 2023-11-14 | 上海淇毓信息科技有限公司 | Dynamic global level grouping method, device and system based on financial clients |
| CN110222030A (en) * | 2019-05-13 | 2019-09-10 | 福建天泉教育科技有限公司 | The method of Database Dynamic dilatation, storage medium |
| CN111199386A (en) * | 2019-12-27 | 2020-05-26 | 天阳宏业科技股份有限公司 | Workflow engine and implementation method thereof |
| CN112184204A (en) * | 2020-09-30 | 2021-01-05 | 银盛支付服务股份有限公司 | Data operation method and system based on data routing rule and database and table division |
| CN117473071A (en) * | 2023-12-27 | 2024-01-30 | 珠海格力电器股份有限公司 | Data retrieval method, device, equipment and computer readable medium |
| CN117473071B (en) * | 2023-12-27 | 2024-04-05 | 珠海格力电器股份有限公司 | Data retrieval method, device, equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107229688A (en) | A kind of database level point storehouse point table method and system, server | |
| Venkatraman et al. | SQL versus NoSQL movement with big data analytics | |
| AU2014207599B2 (en) | Efficient query processing using histograms in a columnar database | |
| US9767427B2 (en) | Modeling multi-dimensional sequence data over streams | |
| CN103577440B (en) | A kind of data processing method and device in non-relational database | |
| CN104199986A (en) | Vector data space indexing method base on hbase and geohash | |
| WO2012012968A1 (en) | Data partitioning method for distributed parallel database system | |
| US10114846B1 (en) | Balanced distribution of sort order values for a multi-column sort order of a relational database | |
| CN107423422A (en) | Spatial data distributed storage and search method and system based on grid | |
| CN107133342A (en) | A kind of IndexR real-time data analysis storehouse | |
| CN103631912B (en) | A kind of method utilizing non-relational database storage magnanimity monitoring industrial equipment data | |
| WO2016149552A1 (en) | Compaction policy | |
| CN105408857A (en) | Generating a multi-column index for relational databases by interleaving data bits for selectivity | |
| CN107357846A (en) | The methods of exhibiting and device of relation map | |
| US9189489B1 (en) | Inverse distribution function operations in a parallel relational database | |
| CN105631003A (en) | Intelligent index establishing, inquiring and maintaining method supporting mass data classification and counting | |
| CN102737123B (en) | A kind of multidimensional data distribution method | |
| CN104298779A (en) | Processing method and system for massive data processing | |
| CN108009290A (en) | A kind of data modeling and storage method of track traffic command centre gauze big data | |
| CN111708895B (en) | Knowledge graph system construction method and device | |
| CN104991741B (en) | A kind of situation adaptation power network big data storage method based on key-value model | |
| CN105786877B (en) | A kind of date storage method, system and querying method | |
| CN113127741B (en) | Cache method for reading and writing data of mass users and posts in part-time post recommendation system | |
| Lwin et al. | Non-redundant dynamic fragment allocation with horizontal partition in Distributed Database System | |
| CN106528448A (en) | Distributed caching mechanism for multi-source heterogeneous electronic commerce big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 200080 room 1050, No. 901-908 East Daming Road, Shanghai, Hongkou District Applicant after: Shanghai pre long Mdt InfoTech Ltd Address before: 200080 room 1050, No. 901-908 East Daming Road, Shanghai, Hongkou District Applicant before: Shanghai long ago Financial Information Service Co., Ltd. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20210223 |