CN110955664A - Method and device for routing messages in different banks and tables - Google Patents
Method and device for routing messages in different banks and tables Download PDFInfo
- Publication number
- CN110955664A CN110955664A CN201911220228.0A CN201911220228A CN110955664A CN 110955664 A CN110955664 A CN 110955664A CN 201911220228 A CN201911220228 A CN 201911220228A CN 110955664 A CN110955664 A CN 110955664A
- Authority
- CN
- China
- Prior art keywords
- sub
- database
- message routing
- configuration information
- routing configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 239000012634 fragment Substances 0.000 claims abstract description 41
- 238000012545 processing Methods 0.000 claims abstract description 26
- 230000015654 memory Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000000638 solvent extraction Methods 0.000 claims description 2
- 238000013467 fragmentation Methods 0.000 description 14
- 238000006062 fragmentation reaction Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 241000533950 Leucojum Species 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method and a device for routing database and table division messages, wherein the method comprises the following steps: determining a logical table name and a database and table dividing message routing key value in a received database operation request; determining a sub-library sub-table message routing configuration information list corresponding to the logical table name, wherein the routing configuration information of each sub-library sub-table message comprises: a fragment number, a database name, and a logical table name; determining a fragment number according to the database and table dividing message routing key value; and determining corresponding sub-library sub-table message routing configuration information from the sub-library sub-table message routing configuration information list according to the calculated fragment number, and routing the database operation request to a database corresponding to the corresponding sub-library sub-table message routing configuration information for processing. The method and the device for routing the messages in the sub-base and the sub-table improve the processing efficiency of the application service and reduce the risk and pressure of a single base of a system.
Description
Technical Field
The invention relates to the field of databases, in particular to a database-based and table-based message routing method and device.
Background
Data is used for a long time when calculating the risk indicator in the risk performance field, for example, the performance of the performance fund is used for all data from establishment to calculation. There are also a number of similar situations that result in data that cannot be conventionally segmented into current and historical data. With the development of time and service, the data amount in the table will be larger and larger, and accordingly, the overhead of data operation, addition, deletion, modification and check will be larger and larger. In addition, resources (CPU, disk, memory, IO, etc.) of one server are limited, and finally, the data volume and data processing capability that can be carried by the database will suffer from bottlenecks.
Disclosure of Invention
In order to solve at least one of the above technical problems, the present invention provides a method and an apparatus for routing a database-table-splitting message.
In order to achieve the above object, according to an aspect of the present invention, there is provided a method for routing a database and table-divided message, the method including:
determining a logical table name and a database and table dividing message routing key value in a received database operation request;
determining a sub-library and sub-table message routing configuration information list corresponding to the logical table name, wherein the sub-library and sub-table message routing configuration information list comprises a plurality of pieces of sub-library and sub-table message routing configuration information, and each piece of sub-library and sub-table message routing configuration information comprises: a fragment number, a database name, and a logical table name;
determining a fragment number according to the database and table dividing message routing key value;
and determining corresponding sub-library sub-table message routing configuration information from the sub-library sub-table message routing configuration information list according to the calculated fragment number, and routing the database operation request to a database corresponding to the corresponding sub-library sub-table message routing configuration information for processing.
Optionally, before the determining the sub-library and sub-table message routing configuration information list corresponding to the logical table name, the method further includes:
classifying all the sub-base sub-table message routing configuration information according to the logical table names to obtain a sub-base sub-table message routing configuration information list corresponding to each logical table name;
and loading the sub-library and sub-table message routing configuration information list corresponding to each logic table name into a cache.
Optionally, the determining the sub-library and sub-table message routing configuration information list corresponding to the logical table name includes:
and searching a database sub-table message routing configuration information list corresponding to the logical table name of the database operation request from the cache.
Optionally, the determining the segment number according to the store-splitting table-splitting message routing key value includes:
calculating the hash value of the database and table dividing message routing key value;
and determining a fragment number corresponding to the database and table dividing message routing key value according to the position of the Hash value of the database and table dividing message routing key value on a preset Hash ring.
In order to achieve the above object, according to another aspect of the present invention, there is provided a database-table message routing apparatus, including:
the database operation request processing unit is used for determining the logical table name and the database partitioning and table dividing message routing key value in the received database operation request;
a database-based and table-based message routing configuration information list determining unit, configured to determine a database-based and table-based message routing configuration information list corresponding to the logical table name, where the database-based and table-based message routing configuration information list includes a plurality of pieces of database-based and table-based message routing configuration information, and each piece of database-based and table-based message routing configuration information includes: a fragment number, a database name, and a logical table name;
the fragment number calculation unit is used for determining a fragment number according to the library and table division message routing key value;
and the database operation request routing unit is used for determining corresponding sub-database sub-table message routing configuration information from the sub-database sub-table message routing configuration information list according to the calculated fragment number, and routing the database operation request to a database corresponding to the corresponding sub-database sub-table message routing configuration information for processing.
Optionally, the database-splitting and table-splitting message routing apparatus further includes:
the database sub-table message routing configuration information classifying unit is used for classifying all database sub-table message routing configuration information according to the logical table names to obtain a database sub-table message routing configuration information list corresponding to each logical table name;
and the cache unit is used for loading the sub-library and sub-table message routing configuration information list corresponding to each logic table name into the cache.
Optionally, the database-based and table-divided message routing configuration information list determining unit is specifically configured to search the cache for the database-based and table-divided message routing configuration information list corresponding to the logical table name of the database operation request.
Optionally, the slice number calculating unit includes:
the hash calculation module is used for calculating the hash value of the database and table dividing message routing key value;
and the fragment number determining module is used for determining the fragment number corresponding to the database and table dividing message routing key value according to the position of the Hash value of the database and table dividing message routing key value on a preset Hash ring.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the above library-based and table-divided message routing method when executing the computer program.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed in a computer processor, implements the steps in the above-described banking and table-splitting message routing method.
The invention has the beneficial effects that: the embodiment of the invention routes messages such as database operation requests and the like in a database cluster comprising a plurality of sub-libraries according to set sub-library and sub-table message routing configuration information, wherein the sub-library and sub-table message routing configuration information is generated according to the fragment number, the logic table and the information of the physical table in each sub-library.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts. In the drawings:
FIG. 1 is a first flowchart of a method for routing a database-based and table-based message according to an embodiment of the present invention;
FIG. 2 is a second flowchart of a method for routing a database-based and table-based message according to an embodiment of the present invention;
FIG. 3 is a flow chart of the embodiment of the present invention for calculating a tile number;
fig. 4 is a first block diagram of a message routing apparatus according to an embodiment of the present invention;
fig. 5 is a second structural block diagram of the database-based and table-based message routing apparatus according to the embodiment of the present invention;
FIG. 6 is a block diagram of a block number calculation unit according to an embodiment of the present invention;
FIG. 7 is a flow chart of message routing according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a cross-library processing scheme according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of a computer apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It should be noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of the present invention and the above-described drawings, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
Some technical words in the present application are explained below:
database and table division: the database and table division is a common data distributed processing mode, and the main database and table division strategies are as follows: vertical cutting: splitting the data table by taking the column as a unit, reducing the coupling degree between service logics, and horizontally splitting: the data table is split according to the row unit, and the data volume of the single table is reduced;
logic table: the same logic and data structure table for a horizontally split database (table). Example (c): splitting order data into 10 tables according to the main key mantissas, wherein the tables are t _ order _0 to t _ order _9, and the logical table names of the tables are t _ order;
physical table: physical tables that actually exist in the sharded database. I.e., t _ order _0 to t _ order _9 in the logical table split example;
a data node: the smallest unit of data fragmentation. Consists of data source names and data tables, example: ds0.t _ order _ 0;
broadcasting table: refers to the tables existing in all the sharded data sources, and the table structure and the data in the tables are completely consistent in each database. The method is suitable for a scene with small data volume and needing to be associated with a table of mass data for query, for example: a dictionary table;
warehouse and table dividing key: the database field for sharding is a key field for horizontally splitting a database (table). Example (c): modulus segmentation is carried out on the mantissa of the order main key in the order table, and the order main key is a segmentation field;
distributed transaction: it means that the participants of the transaction, the servers supporting the transaction, the resource server and the transaction manager are respectively located on different nodes.
Fig. 1 is a first flowchart of a library-based table-divided message routing method according to an embodiment of the present invention, and as shown in fig. 1, the library-based table-divided message routing method according to the embodiment includes steps S101 to S104.
Step S101, determining the logical table name and the database-dividing and table-dividing message routing key value in the received database operation request.
In the embodiment of the present invention, the database-to-table message routing key is a value of a preset field in the database operation request.
In the embodiment of the invention, the existing single database is divided into a database cluster comprising a plurality of databases through database division and table division, and the database cluster comprises two types of databases: performing bank division: the sub-banks are provided with a plurality of sub-banks from 1 to n and are used for storing data of an amplification data table (T _0 table) and a broadcast table (a table which needs to be frequently associated, such as G _ TBL); default library: and storing a configuration class table (database and table dividing message routing configuration information, configuration information C _ TBL and basic information R _ TBL), a broadcast table, statistical result table data calculated according to database dividing data and the like. And then setting the information routing configuration information of the sub-database and sub-table according to the condition of the sub-database and sub-table, and when receiving the database operation request, routing the database operation request to a corresponding physical table in the sub-database for processing according to the logical table name and the information routing key value of the sub-database and sub-table in the database operation request and the preset information routing configuration information of the sub-database and sub-table.
In an optional embodiment of the present invention, the database operation request may adopt an existing database language SQL, where the database operation request SQL includes database operation fields such as a logical table name. In the embodiment of the present invention, the database operation request SQL further includes a combined ID field, where the combined ID field is a store-dividing and table-dividing message routing key, and a value corresponding to the combined ID field is a store-dividing and table-dividing message routing key value. In an alternative embodiment of the present invention, the combined ID field may be a field preset in the database operation request SQL.
Step S102, determining a sub-library sub-table message routing configuration information list corresponding to the logical table name, wherein the sub-library sub-table message routing configuration information list comprises a plurality of sub-library sub-table message routing configuration information, and each sub-library sub-table message routing configuration information comprises: a tile number, a database name, and a logical table name.
In an optional embodiment of the present invention, the database-based table-divided message routing configuration information may refer to table 1, and as shown in the embodiment of table 1, each database-based table-divided message routing configuration information includes: the method comprises the following steps of information such as a fragment number, a database name, a logical table name, a physical table name, a fragment mode and the like, wherein the database name corresponds to the name of a sub-library.
In this embodiment of the present invention, in this step, all the sub-pool and sub-table message routing configuration information may be summarized to form a total sub-pool and sub-table message routing configuration information list as shown in table 1 below, and the total sub-pool and sub-table message routing configuration information list is further classified according to the logical table names, so as to obtain a sub-pool and sub-table message routing configuration information list corresponding to each logical table name. And further, after the logical table name in the database operation request is determined, the sub-database sub-table message routing configuration information list corresponding to the logical table name can be determined.
| Number of the section | Name of database | Logical table name | Physical table name | Slicing mode |
| bk1 | DB0 | user_info | user_info1 | split_db_table |
| bk2 | DB0 | user_info | user_info2 | split_db_table |
| bk3 | DB1 | user_info | user_info3 | split_db_table |
| bk4 | DB1 | user_info | user_info4 | split_db_table |
| 1 | DB0 | org_info | org_info1 | split_db |
| 2 | DB1 | org_info | org_info2 | split_db |
| 1 | DB0 | role_info | role_info1 | split_table |
| 2 | DB0 | role_info | role_info2 | split_table |
TABLE 1
And step S103, determining the fragment number according to the library and table dividing message routing key value.
In an optional embodiment of the present invention, this step may determine the fragment number by using a plurality of methods, and in an optional embodiment of the present invention, this step may first establish a correspondence between different store-dividing and table-dividing message routing key values and the fragment number, and then directly determine the corresponding fragment number after obtaining the store-dividing and table-dividing message routing key value. In other alternative embodiments of the present invention, the present step may also adopt the existing slice number calculation method.
And step S104, determining corresponding sub-library sub-table message routing configuration information from the sub-library sub-table message routing configuration information list according to the calculated fragment number, and routing the database operation request to a database corresponding to the corresponding sub-library sub-table message routing configuration information for processing.
In an optional embodiment of the present invention, in this step, one or more pieces of library-splitting table-splitting message routing configuration information corresponding to the splitting number may be determined from the library-splitting table-splitting message routing configuration information list obtained in step S102 according to the splitting number obtained in step S103, so as to read a database name and a physical table name in the library-splitting table-splitting message routing configuration information, and finally, the database operation request is routed to the library corresponding to the database name, and corresponding operation processing is performed in the physical table corresponding to the physical table name.
As can be seen from the above description, the database-based and table-based message routing method according to the embodiment of the present invention improves the processing efficiency of the application service and reduces the risk and pressure of the system database by performing database-based and table-based splitting on the single database and routing messages such as the database operation request according to the database-based and table-based message routing configuration information. The embodiment of the invention considers that various basic data required by calculation are stored by adopting a mode of database-by-database and table-by-table message routing, thereby ensuring normal and rapid processing of services.
Fig. 2 is a second flowchart of the database-based and table-divided message routing method according to the embodiment of the present invention, and as shown in fig. 2, the database-based and table-divided message routing method further includes steps S201 to S202.
Step S201, all the sub-base sub-table message routing configuration information is classified according to the logic table names, and a sub-base sub-table message routing configuration information list corresponding to each logic table name is obtained.
In this embodiment of the present invention, in this step, all the sub-pool and sub-table message routing configuration information may be summarized to form a total sub-pool and sub-table message routing configuration information list as shown in table 1 below, and the total sub-pool and sub-table message routing configuration information list is further classified according to the logical table names, so as to obtain a sub-pool and sub-table message routing configuration information list corresponding to each logical table name.
In an optional embodiment of the present invention, all the sub-pool sub-table message routing configuration information is stored in the default pool, in this step, the sub-pool sub-table message routing configuration information in the default pool is queried, each piece of configuration information is cycled and converted into a map form, key is a logical table name, and value is a list of all sub-pool sub-table message routing configuration information under the corresponding logical table name.
Step S202, loading the sub-library and sub-table message routing configuration information list corresponding to each logic table name into a cache.
In the embodiment of the invention, all the inquired configuration information is cached in the redis cache in a key-value mode, and when the message routing is carried out, the corresponding configuration information is directly obtained from the cache, so that the routing efficiency is improved.
In an optional embodiment of the present invention, the determining of the sub-repository sub-table message routing configuration information list corresponding to the logical table name in step S102 is specifically to search the sub-repository sub-table message routing configuration information list corresponding to the logical table name of the database operation request from the cache. According to the embodiment of the invention, the database sub-table message routing configuration information corresponding to each logic table name is obtained from the cache, so that the database pressure is reduced, and the message routing speed is higher.
Fig. 3 is a flowchart of calculating a segment number according to an embodiment of the present invention, and as shown in fig. 3, in an alternative embodiment of the present invention, the determining a segment number according to the store-splitting table message routing key value in step S103 specifically includes step S301 and step S302.
Step S301, calculating the hash value of the database and table dividing message routing key value.
Step S302, determining a segment number corresponding to the sub-warehouse and sub-table message routing key value according to the position of the Hash value of the sub-warehouse and sub-table message routing key value on a preset Hash ring.
When the embodiment of the invention carries out message routing, a corresponding database sub-table message routing configuration information list is obtained according to the name of the logic table, if the information list exists, consistent Hash processing is carried out according to the database sub-table message routing key value, the corresponding fragment number is calculated, and then specific database sub-table message routing configuration information is obtained according to the fragment number, so that the corresponding database operation request can be routed to the corresponding database and physical table for processing.
In an optional embodiment of the present invention, a consistent Hash processing method may be adopted to calculate the fragmentation number, and a specific fragmentation policy may be: and calculating the fragmentation node according to the physical table name and the fragmentation number through a Hash function, and distributing the fragmentation node obtained by each physical table name and the fragmentation number to a position (fragmentation node) on a Hash ring of 2^ 32. In addition, each sub-pool sub-table message routing key is also mapped to a location on the 2^32 Hash ring. The database and table splitting message routing key value is finally placed in the fragmentation node which is closest to the position of the database and has the position number larger than or equal to the value, namely placed in the next clockwise fragmentation node.
In an optional embodiment of the invention, the fragmentation strategy adopts a Google MurMurHash consistency Hash algorithm, the data distribution is uniform, and the maximum and minimum data amount of each fragment are within 2 percent. In the optional embodiment of the invention, each sub-library and sub-table message routing key value is dispersed on a Hash ring of 2^32 according to the key value (key) calculated by a Hash algorithm, and then the key value obtains the clockwise nearest fragmentation node number, namely the fragmentation number corresponding to the sub-library and sub-table message routing key value.
Fig. 7 is a flowchart of message routing according to an embodiment of the present invention, and as shown in fig. 7, in an alternative embodiment of the present invention, the process flow of sub-base and sub-table routing may be:
1. acquiring a logical table name, acquiring a corresponding logical table name from a standard fragmentation interface of Shardingsphere, and configuring in a sub-library sub-table configuration file, wherein if the logical table name is not configured, the logical table name cannot be acquired;
2. and acquiring a corresponding database sub-table message routing configuration information list according to the logical table name. Obtaining a database sub-table message routing configuration information list corresponding to the logic table from a redis cache, if the database sub-table message routing configuration information list does not exist, querying a default database, and if the configuration of the logic table exists in the database, adding the default database into the redis cache;
3. and judging whether a database-splitting table-splitting message routing configuration information list of the logic table exists or not, if not, directly returning to a default database and the logic table name, and if so, continuing.
Fig. 8 is a schematic diagram of a cross-repository processing scheme according to an embodiment of the present invention, and as shown in fig. 8, the present invention sets up a quartic repository process among multiple sub-repositories, and an embodiment of the present invention may split a cross-repository transaction into local transactions of multiple sub-repositories, and then sequentially switch data sources to execute each sub-local transaction in each sub-repository. Recording all transaction operations SQL by using a transaction log table, and deleting the transaction log if the sub-transaction is successfully submitted; if the execution fails, the resubmission is tried according to the configured retry number, namely, the best-effort resubmission is carried out, the consistency of the data is guaranteed as much as possible, and here, synchronous retry or asynchronous retry can be adopted according to different business scenarios, balance C and A. Certainly, when the application layer avoids generating cross-library transactions as much as possible, for example, batch processing, the corresponding library may be determined according to the combination ID, and then a plurality of combinations in the same database are combined into one batch for processing, so as to effectively avoid generating cross-library transactions.
In an optional embodiment of the invention, each message in the database-splitting and table-splitting message routing method has a global unique ID, different types of global unique IDs can be generated according to different incoming generation types, the global unique ID is generated in a mode of supporting a library number, a component number, a date and a cyclic serial number, and the global unique ID is generated by a snowflake algorithm.
In the optional embodiment of the present invention, the generation manner of obtaining the unique ID may have three values: SNOW _ flag (snowflake algorithm), CHAR _20(20 bit fixed length string), USER _ default (USER defined), all algorithms implement a unified interface method. In an optional embodiment of the present invention, if the generation mode is a SNOW _ flag (snowflake algorithm), a snowflake algorithm with a twitter open source is adopted, and no third-party component is required to be relied on, so that the extensibility and maintainability are maximally simplified, but attention is required (strong dependence on time, and repeated IDs are generated if a clock dials back). If the generation mode is CHAR-20, generating 20-bit fixed-length character string global unique ID according to the current database number, application component number, 8-bit date and 10-bit cyclic serial number, wherein the maximum data volume is 10 hundred million per day. And if the generation mode is USER _ default, calling the corresponding interface method to generate the global unique ID according to the incoming bearer ID.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
Based on the same inventive concept, the embodiment of the present invention further provides a database-based and table-based message routing apparatus, which can be used to implement the database-based and table-based message routing method described in the foregoing embodiment, as described in the following embodiments. Because the principle of the database-based and table-based message routing device for solving the problems is similar to the database-based and table-based message routing method, the embodiment of the database-based and table-based message routing device can refer to the embodiment of the database-based and table-based message routing method, and repeated parts are not described again. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 4 is a first structural block diagram of a database-splitting and table-splitting message routing apparatus according to an embodiment of the present invention, and as shown in fig. 4, the database-splitting and table-splitting message routing apparatus according to the embodiment of the present invention includes: the system comprises a database operation request processing unit 1, a database sub-table information routing configuration information list determining unit 2, a fragment number calculating unit 3 and a database operation request routing unit 4.
The database operation request processing unit 1 is configured to determine a logical table name and a store-dividing and table-dividing message routing key in the received database operation request. And the database and table dividing message routing key value is a value of a preset field in the database operation request.
A sub-library and sub-table message routing configuration information list determining unit 2, configured to determine a sub-library and sub-table message routing configuration information list corresponding to the logical table name, where the sub-library and sub-table message routing configuration information list includes a plurality of pieces of sub-library and sub-table message routing configuration information, and each piece of sub-library and sub-table message routing configuration information includes: a tile number, a database name, and a logical table name.
And the fragment number calculating unit 3 is used for determining the fragment number according to the library-based and table-divided message routing key value.
And the database operation request routing unit 4 is configured to determine, according to the calculated fragment number, corresponding sub-library sub-table message routing configuration information from the sub-library sub-table message routing configuration information list, and route the database operation request to a database corresponding to the corresponding sub-library sub-table message routing configuration information for processing.
Fig. 5 is a second structural block diagram of the database-based and table-based message routing apparatus according to the embodiment of the present invention, and as shown in fig. 5, the database-based and table-based message routing apparatus according to the embodiment of the present invention further includes: a database and table dividing message routing configuration information classification unit 5 and a cache unit 6.
And the database-based and table-based message routing configuration information classifying unit 5 is configured to classify all database-based and table-based message routing configuration information according to the logical table names to obtain a database-based and table-based message routing configuration information list corresponding to each logical table name.
And the cache unit 6 is configured to load the sub-library and sub-table message routing configuration information list corresponding to each logical table name into the cache.
In an optional embodiment of the present invention, the sub-pool sub-table message routing configuration information list determining unit 2 may search the sub-pool sub-table message routing configuration information list corresponding to the logical table name of the database operation request from the cache.
Fig. 6 is a block diagram of a structure of a slice number calculating unit according to an embodiment of the present invention, and as shown in fig. 6, in the embodiment of the present invention, the slice number calculating unit 3 includes: a hash calculation module 301 and a slice number determination module 302.
A hash calculation module 301, configured to calculate a hash value of the store-and-table-splitting message routing key.
A fragment number determining module 302, configured to determine, according to a position of the Hash value of the split-pool and split-table message routing key on a preset Hash ring, a fragment number corresponding to the split-pool and split-table message routing key.
To achieve the above object, according to another aspect of the present application, there is also provided a computer apparatus. As shown in fig. 9, the computer device comprises a memory, a processor, a communication interface and a communication bus, wherein a computer program that can be run on the processor is stored in the memory, and the steps of the method of the embodiment are realized when the processor executes the computer program.
The processor may be a Central Processing Unit (CPU). The Processor may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or a combination thereof.
The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and units, such as the corresponding program units in the above-described method embodiments of the present invention. The processor executes various functional applications of the processor and the processing of the work data by executing the non-transitory software programs, instructions and modules stored in the memory, that is, the method in the above method embodiment is realized.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor, and the like. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and such remote memory may be coupled to the processor via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more units are stored in the memory and when executed by the processor perform the method of the above embodiments.
The specific details of the computer device may be understood by referring to the corresponding related descriptions and effects in the above embodiments, and are not described herein again.
To achieve the above object, according to another aspect of the present application, there is also provided a computer-readable storage medium storing a computer program which, when executed in a computer processor, implements the steps in the above-described banking and table-splitting message routing method. It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD) or a Solid State Drive (SSD), etc.; the storage medium may also comprise a combination of memories of the kind described above.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for routing database and table messages, comprising:
determining a logical table name and a database and table dividing message routing key value in a received database operation request;
determining a sub-library and sub-table message routing configuration information list corresponding to the logical table name, wherein the sub-library and sub-table message routing configuration information list comprises a plurality of pieces of sub-library and sub-table message routing configuration information, and each piece of sub-library and sub-table message routing configuration information comprises: a fragment number, a database name, and a logical table name;
determining a fragment number according to the database and table dividing message routing key value;
and determining corresponding sub-library sub-table message routing configuration information from the sub-library sub-table message routing configuration information list according to the calculated fragment number, and routing the database operation request to a database corresponding to the corresponding sub-library sub-table message routing configuration information for processing.
2. The method according to claim 1, further comprising, before said determining the table of the sub-pool and sub-table message routing configuration information corresponding to the logical table name:
classifying all the sub-base sub-table message routing configuration information according to the logical table names to obtain a sub-base sub-table message routing configuration information list corresponding to each logical table name;
and loading the sub-library and sub-table message routing configuration information list corresponding to each logic table name into a cache.
3. The method according to claim 2, wherein determining the sub-repository sub-table message routing configuration information list corresponding to the logical table name comprises:
and searching a database sub-table message routing configuration information list corresponding to the logical table name of the database operation request from the cache.
4. The method of claim 1, wherein determining a segment number according to the repository-based and table-based message routing key comprises:
calculating the hash value of the database and table dividing message routing key value;
and determining a fragment number corresponding to the database and table dividing message routing key value according to the position of the Hash value of the database and table dividing message routing key value on a preset Hash ring.
5. A database and table splitting message routing apparatus, comprising:
the database operation request processing unit is used for determining the logical table name and the database partitioning and table dividing message routing key value in the received database operation request;
a database-based and table-based message routing configuration information list determining unit, configured to determine a database-based and table-based message routing configuration information list corresponding to the logical table name, where the database-based and table-based message routing configuration information list includes a plurality of pieces of database-based and table-based message routing configuration information, and each piece of database-based and table-based message routing configuration information includes: a fragment number, a database name, and a logical table name;
the fragment number calculation unit is used for determining a fragment number according to the library and table division message routing key value;
and the database operation request routing unit is used for determining corresponding sub-database sub-table message routing configuration information from the sub-database sub-table message routing configuration information list according to the calculated fragment number, and routing the database operation request to a database corresponding to the corresponding sub-database sub-table message routing configuration information for processing.
6. The device for routing messages according to claim 5, further comprising:
the database sub-table message routing configuration information classifying unit is used for classifying all database sub-table message routing configuration information according to the logical table names to obtain a database sub-table message routing configuration information list corresponding to each logical table name;
and the cache unit is used for loading the sub-library and sub-table message routing configuration information list corresponding to each logic table name into the cache.
7. The device according to claim 6, wherein the database-based and table-based message routing configuration information list determining unit is specifically configured to search the cache for the database-based and table-based message routing configuration information list corresponding to the logical table name of the database operation request.
8. The device according to claim 5, wherein the fragment number calculation unit comprises:
the hash calculation module is used for calculating the hash value of the database and table dividing message routing key value;
and the fragment number determining module is used for determining the fragment number corresponding to the database and table dividing message routing key value according to the position of the Hash value of the database and table dividing message routing key value on a preset Hash ring.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 4 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed in a computer processor, carries out the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911220228.0A CN110955664A (en) | 2019-12-03 | 2019-12-03 | Method and device for routing messages in different banks and tables |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911220228.0A CN110955664A (en) | 2019-12-03 | 2019-12-03 | Method and device for routing messages in different banks and tables |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110955664A true CN110955664A (en) | 2020-04-03 |
Family
ID=69979475
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911220228.0A Pending CN110955664A (en) | 2019-12-03 | 2019-12-03 | Method and device for routing messages in different banks and tables |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110955664A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102890651A (en) * | 2011-07-19 | 2013-01-23 | 阿里巴巴集团控股有限公司 | Method and device for testing scene data |
| CN104111924A (en) * | 2013-04-16 | 2014-10-22 | 中国移动通信集团广东有限公司 | Database system |
| CN105930387A (en) * | 2016-04-14 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Data operation system and method based on data routing and sharding |
| CN106055587A (en) * | 2016-05-21 | 2016-10-26 | 乐视控股（北京）有限公司 | Partitioning database system and routing method thereof |
| WO2017012492A1 (en) * | 2015-07-22 | 2017-01-26 | 阿里巴巴集团控股有限公司 | Form identifier generation method, form shunting method and apparatus |
| CN107085570A (en) * | 2016-02-14 | 2017-08-22 | 华为技术有限公司 | Data processing method, application server and router |
-
2019
- 2019-12-03 CN CN201911220228.0A patent/CN110955664A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102890651A (en) * | 2011-07-19 | 2013-01-23 | 阿里巴巴集团控股有限公司 | Method and device for testing scene data |
| CN104111924A (en) * | 2013-04-16 | 2014-10-22 | 中国移动通信集团广东有限公司 | Database system |
| WO2017012492A1 (en) * | 2015-07-22 | 2017-01-26 | 阿里巴巴集团控股有限公司 | Form identifier generation method, form shunting method and apparatus |
| CN107085570A (en) * | 2016-02-14 | 2017-08-22 | 华为技术有限公司 | Data processing method, application server and router |
| CN105930387A (en) * | 2016-04-14 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Data operation system and method based on data routing and sharding |
| CN106055587A (en) * | 2016-05-21 | 2016-10-26 | 乐视控股（北京）有限公司 | Partitioning database system and routing method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2019165665A1 (en) | Domain name resolution method, server and system | |
| CN107085570B (en) | Data processing method, application server and router | |
| AU2014212780A1 (en) | Data stream splitting for low-latency data access | |
| CN110019080B (en) | Data access method and device | |
| WO2018161881A1 (en) | Structuralized data processing method, data storage medium, and computer apparatus | |
| WO2018036549A1 (en) | Distributed database query method and device, and management system | |
| US8015195B2 (en) | Modifying entry names in directory server | |
| WO2022111313A1 (en) | Request processing method and micro-service system | |
| CN111159219B (en) | Data management method, device, server and storage medium | |
| CN110727738B (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| CN106909556B (en) | Memory cluster storage balancing method and device | |
| CN108399175B (en) | Data storage and query method and device | |
| CN113157734B (en) | Data processing method, device and equipment based on search framework and storage medium | |
| CN106909557B (en) | Memory cluster storage method and device and memory cluster reading method and device | |
| CN110795419A (en) | Method and device for dynamic database-based routing | |
| CN114116777A (en) | Data processing method, device, equipment and storage medium | |
| CN111858586B (en) | Data processing method and device | |
| CN112765169A (en) | Data processing method, device, equipment and storage medium | |
| CN117009389A (en) | Data caching method, device, electronic equipment and readable storage medium | |
| CN116775712A (en) | Method, device, electronic equipment, distributed system and storage medium for inquiring linked list | |
| CN110955664A (en) | Method and device for routing messages in different banks and tables | |
| CN112148728A (en) | Method, apparatus and computer program product for information processing | |
| US20190079987A1 (en) | Distributed data storage | |
| JP2016045594A (en) | Data processing apparatus, data processing method, and data processing program | |
| CN110705935B (en) | Logistics document processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220914 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200403 |
|
| RJ01 | Rejection of invention patent application after publication |