CN111291112B - Read-write control method and device for distributed database and electronic equipment - Google Patents
Read-write control method and device for distributed database and electronic equipment Download PDFInfo
- Publication number
- CN111291112B CN111291112B CN201811496200.5A CN201811496200A CN111291112B CN 111291112 B CN111291112 B CN 111291112B CN 201811496200 A CN201811496200 A CN 201811496200A CN 111291112 B CN111291112 B CN 111291112B
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- redistribution
- target node
- old
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention provides a read-write control method and device of a distributed database and electronic equipment. The method comprises the following steps: establishing a redistribution table; modifying the redistribution table according to a data table to be redistributed, wherein the data table is locked when the modification of the content corresponding to the data table in the redistribution table is started; unlocking the data table when the modification is completed; and performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table. According to the embodiment of the invention, the read-write control of the data table is performed through the pre-constructed redistribution table, the corresponding data table in the database is only required to be locked before the data is redistributed in the database and in the process of modifying the redistribution table, and the data table is not required to be locked when the data is redistributed in the database, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
Description
Technical Field
The present invention relates to the field of distributed databases, and in particular, to a method and an apparatus for controlling reading and writing of a distributed database, and an electronic device.
Background
In the distributed database, when the performance or capacity of the node cannot meet the service requirement, the capacity of the database needs to be expanded; similarly, if the business is withered, the database can be also scaled; in addition, when the data amount of some nodes is particularly large, the problem of data skew can be solved by modifying the distribution key. In the process of data redistribution such as expanding, shrinking and modifying the distribution key, in order to ensure the consistency of data, the data table being redistributed needs to be locked, and the read-write operation cannot be performed on the data table until the redistribution of the data table is completed.
In the process of implementing the present invention, the inventors have found that at least the following problems exist in the prior art: in the prior art, the whole process of continuously redistributing the data table is required to be locked, so that long-time data blocking is caused, the performance of the database is deteriorated, and the normal operation of related services is influenced.
Disclosure of Invention
The embodiment of the invention provides a read-write control method and device for a distributed database and electronic equipment, which are used for solving the defect of low acquisition efficiency of training data in the prior art.
To achieve the above objective, an embodiment of the present invention provides a method for controlling reading and writing of a distributed database, including:
establishing a corresponding redistribution table according to a data table, wherein the redistribution table at least comprises a state identifier of the data table, redistribution data and target node determining data, the state identifier is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table;
modifying the redistribution table according to a data table to be redistributed, wherein the data table is locked when the modification of the content corresponding to the data table in the redistribution table is started;
when the modification of the content corresponding to the data table in the redistribution table is completed, unlocking the data table;
and performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table.
The embodiment of the invention also provides a read-write control device of the distributed database, which comprises:
the system comprises a redistribution table establishing module, a redistribution table determining module and a data processing module, wherein the redistribution table establishing module is used for establishing a corresponding redistribution table according to a data table, the redistribution table at least comprises a state identifier of the data table, redistribution data and target node determining data, the state identifier is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table;
The preprocessing module is used for modifying the redistribution table according to the data table to be redistributed, wherein the data table is locked when the content corresponding to the data table in the redistribution table is modified, and the data table is unlocked when the content corresponding to the data table in the redistribution table is modified;
and the read-write control module is used for performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table.
The embodiment of the invention also provides electronic equipment, which comprises:
a memory for storing a program;
a processor for running the program stored in the memory for:
establishing a corresponding redistribution table according to a data table, wherein the redistribution table at least comprises a state identifier of the data table, redistribution data and target node determining data, the state identifier is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table;
modifying the redistribution table according to a data table to be redistributed, wherein the data table is locked when the modification of the content corresponding to the data table in the redistribution table is started;
When the modification of the content corresponding to the data table in the redistribution table is completed, unlocking the data table;
and performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table.
According to the read-write control method and device for the distributed database and the electronic equipment, the read-write control of the data table is performed through the pre-built redistribution table, the corresponding data table in the database is only required to be locked before the data is redistributed by the database and in the process of modifying the redistribution table, and the data table is not required to be locked when the data is redistributed by the database, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
Fig. 1 is a system block diagram of a service system provided in an embodiment of the present invention;
FIG. 2 is a flowchart of an embodiment of a method for controlling read/write of a distributed database according to the present invention;
FIG. 3 is a flowchart of another embodiment of a method for controlling reading and writing of a distributed database according to the present invention;
FIG. 4 is a flowchart of a method for controlling reading and writing of a distributed database according to another embodiment of the present invention;
FIG. 5 is a schematic diagram illustrating an embodiment of a read-write control device for a distributed database according to the present invention;
FIG. 6 is a schematic diagram of another embodiment of a read-write control device for a distributed database according to the present invention;
fig. 7 is a schematic structural diagram of an embodiment of an electronic device according to the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In the prior art, in order to ensure the consistency of data in the data redistribution process such as expanding, shrinking and modifying a distribution key of a distributed database, a data table being redistributed needs to be locked, and cannot be read or written until the data table is redistributed, so that long-time data blockage is caused, the performance of the database is deteriorated, and the normal operation of related services is affected. Therefore, the application provides a read-write control scheme of a distributed database, and the main principle of the read-write control scheme is as follows: a redistribution table is pre-constructed to record the status flag (whether in a redistribution state) of each data table to be redistributed, redistribution data (such as new and old distribution keys of the data table, or new and old node configuration numbers of the database), and target node calculation data (such as node configuration numbers of the database, or distribution keys of the data table). Before the data is redistributed in the database, the data table is modified according to the data packets to be redistributed, and the corresponding data table is locked only in the process of modifying the content corresponding to the data table in the redistribution table (for example, in the redistribution table, each data table can correspond to one data table, then, for one data table to be redistributed, the data content of the corresponding row in the redistribution table is modified, meanwhile, when the data content of one row in the redistribution table is modified, the data table corresponding to the data content of the row needs to be locked, and during the locking period, the data table cannot be read and written). When the data is redistributed in the database, the redistribution table is modified, and the corresponding data tables are unlocked, so that the read-write operation can be controlled according to the redistribution table. According to the scheme, the corresponding data table in the database is only required to be locked before the data is redistributed and when the redistribution table is modified, and the data table is not required to be locked when the data is redistributed in the database, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence on related services is reduced.
The method provided by the embodiment of the invention can be applied to any service system applied to a distributed database. Fig. 1 is a system block diagram of a service system provided by an embodiment of the present invention, and the structure shown in fig. 1 is only one example of a service system to which the technical solution of the present invention can be applied. As shown in fig. 1, the service system includes a read-write control device. The device comprises: the redistribution table establishing module, the preprocessing module and the read-write control module may be used to execute the process flows shown in fig. 2, 3 and 4 described below. In the service system, first, a redistribution table may be established in advance. And then, before the data is redistributed in the database, modifying the redistribution table according to the data table to be redistributed, wherein the data table is locked when the modification of the content corresponding to the data table in the redistribution table is started, and the data table is unlocked when the modification of the content corresponding to the data table in the redistribution table is completed. Here, the redistribution table may include at least a state identifier of the data table, redistribution data, and target node determining data. The state identification is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table. Then, when the database redistributes data, the read-write operation can be performed based on the modified redistribution table according to the read-write instruction aiming at the data table. The corresponding data table in the database is only required to be locked before the data is redistributed in the database and in the process of modifying the redistribution table, and the data table is not required to be locked when the data is redistributed in the database, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
The foregoing embodiments are illustrative of the technical principles and exemplary application frameworks of embodiments of the present invention, and the detailed description of specific technical solutions of the embodiments of the present invention will be further described below by means of a plurality of embodiments.
Example 1
Fig. 2 is a flowchart of an embodiment of a method for controlling read/write of a distributed database according to the present invention, where an execution body of the method may be the service system, a node in the distributed database, or a device or a chip integrated on a node device. The read-write control method of the distributed database provided by the embodiment of the invention can be applied to the data redistribution scenes such as database expansion and contraction, modification of distribution keys and the like. As shown in fig. 2, the read-write control method of the distributed database includes the following steps:
s201, establishing a corresponding redistribution table according to the data table.
In the embodiment of the invention, a redistribution table can be constructed in advance for the distributed database. The redistribution table at least comprises state identification, redistribution data and target node determination data of each data table to be redistributed in the database. The state identification is used for marking whether the data table is in a redistribution state, the redistribution data is the change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table.
S202, modifying the redistribution table according to the data table to be redistributed, wherein the data table is locked when the modification of the content corresponding to the data table in the redistribution table is started.
When a user wants to perform data redistribution operations such as expanding and shrinking capacity, modifying a distribution key and the like on a database, a pre-constructed redistribution table can be modified according to a data table to be redistributed in the database before data redistribution is performed. For example, the state identification of the data table, the old and new distribution keys, the number of old and new node configurations of the database, etc. may be modified. Meanwhile, in modifying the redistribution table, it is necessary to lock the data table corresponding to the modified content, and during the locking, the data table cannot be read and written.
S203, unlocking the data table when the content modification corresponding to the data table in the redistribution table is completed.
S204, according to the read-write instruction aiming at the data table, performing read-write operation based on the modified redistribution table.
In the embodiment of the invention, when the modification of the content corresponding to the data table in the redistribution table is completed, the locking of each data table is released. Therefore, if a read-write command for a certain data table in the database is received, the data table can be read and written based on the modified redistribution table.
According to the read-write control method for the distributed database, the read-write control of the data table is performed through the pre-built redistribution table, the corresponding data table in the database is only required to be locked before the data is redistributed in the database and in the process of modifying the redistribution table, and the data table is not required to be locked when the data is redistributed in the database, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
Example two
Fig. 3 is a flowchart of another embodiment of a read-write control method for a distributed database according to the present invention. The embodiment of the invention can be applied to the application scene of modifying the distribution key. As shown in fig. 3, on the basis of the embodiment shown in fig. 2, the method for controlling reading and writing of a distributed database provided in this embodiment may further include the following steps:
s301, establishing a corresponding redistribution table according to the data table.
S302, modifying a redistribution table according to a data table to be redistributed, wherein the data table is locked when the content corresponding to the data table in the redistribution table is modified; and when the modification of the content corresponding to the data table in the redistribution table is completed, unlocking the data table.
S303, according to a read-write instruction aiming at the data table, based on the state identification, the new distribution key, the old distribution key and the node configuration number of the data table, performing read-write operation on the data table.
In the embodiment of the invention, the redistributed data is a new distributing key and an old distributing key of the data table, and the target node determines the node configuration number of the data for the database (namely, the node number distributed to the database). The new distribution key is a distribution key after the modification of the data table, and the old distribution key is a distribution key before the modification of the data table.
Specifically, when the received read-write instruction is an insert operation instruction, the state identifier of the data table may be determined; when the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a new distribution key and the node configuration number, and writing the inserted data into the new target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the old distributing key and the node configuration number, and writing the inserted data into the old target node.
When the received read-write instruction is a query operation instruction, the state identification of the data table can be judged; when the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a new distribution key and the node configuration number, determining an old target node according to a hash value calculated by a key value corresponding to an old distribution key and the node configuration number, and respectively performing data query operation in the new target node and the old target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the old distributing key and the node configuration number, and performing data query operation in the old target node.
When the received read-write instruction is an update operation instruction, the state identification of the data table can be judged; when the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a new distribution key and the node configuration number, determining an old target node according to a hash value calculated by a key value corresponding to an old distribution key and the node configuration number, and respectively performing data updating operation in the new target node and the old target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the old distributing key and the node configuration number, and performing data updating operation in the old target node.
When the received read-write instruction is a deletion operation instruction, the state identification of the data table can be judged; when the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a new distribution key and the node configuration number, determining an old target node according to a hash value calculated by a key value corresponding to an old distribution key and the node configuration number, and respectively performing data deletion operation in the new target node and the old target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the old distributing key and the node configuration number, and performing data deleting operation in the old target node.
According to the read-write control method for the distributed database, the pre-built redistribution table is applied to the application scene of modifying the distribution key, the data table is read and written based on the state identification of the data table, the new distribution key, the old distribution key and the node configuration number, the corresponding data table in the database is only required to be locked before the data table is modified for modifying the distribution key in the process of modifying the redistribution table, and the data table is not required to be locked when the data table is modified for modifying the distribution key, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
Example III
Fig. 4 is a flowchart of a read-write control method of a distributed database according to another embodiment of the present invention. The embodiment of the invention can be applied to application scenes of database expansion and contraction. As shown in fig. 4, on the basis of the embodiment shown in fig. 2, the method for controlling reading and writing of a distributed database provided in this embodiment may further include the following steps:
s401, establishing a corresponding redistribution table according to the data table.
S402, modifying the redistribution table according to the data table to be redistributed, wherein the data table is locked when the content corresponding to the data table in the redistribution table is modified; and when the modification of the content corresponding to the data table in the redistribution table is completed, unlocking the data table.
S403, according to the read-write instruction aiming at the data table, performing read-write operation on the data table based on the state identification and the distribution key of the data table and the new node configuration number and the old node configuration number of the database.
In the embodiment of the invention, the redistribution data is the new node configuration number and the old node configuration number of the database, and the target node determines that the data is a distribution key of the data table. The new node configuration number is the node number of the database after capacity expansion and contraction, and the old node configuration number is the node number of the database before capacity expansion and contraction.
Specifically, when the received read-write instruction is an insert operation instruction, the state identifier of the data table may be determined; when the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a distribution key and the configuration number of the new node, and writing the inserted data into the new target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the distributing key and the configuration number of the old node, and writing the inserted data into the old target node.
When the received read-write instruction is a query operation instruction, the state identification of the data table can be judged; when the data table is in a redistribution state, determining a new target node according to the hash value calculated by the key value corresponding to the distribution key and the new node configuration number, determining an old target node according to the hash value calculated by the key value corresponding to the distribution key and the old node configuration number, and respectively performing data query operation in the new target node and the old target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the distributing key and the configuration number of the old node, and performing data query operation in the old target node.
When the received read-write instruction is an update operation instruction, the state identification of the data table can be judged; when the data table is in a redistribution state, determining a new target node according to the hash value calculated by the key value corresponding to the distribution key and the new node configuration number, determining an old target node according to the hash value calculated by the key value corresponding to the distribution key and the old node configuration number, and respectively performing data updating operation in the new target node and the old target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the distributing key and the configuration number of the old node, and performing data updating operation in the old target node.
When the received read-write instruction is a deletion operation instruction, the state identification of the data table can be judged; when the data table is in a redistribution state, determining a new target node according to the hash value calculated by the key value corresponding to the distribution key and the new node configuration number, determining an old target node according to the hash value calculated by the key value corresponding to the distribution key and the old node configuration number, and respectively performing data deletion operation in the new target node and the old target node; when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the distributing key and the configuration number of the old node, and performing data deleting operation in the old target node.
In addition, the read-write control method of the distributed database provided by the embodiment of the invention can be also applied to application scenes of moving the historical data. Specifically, the redistribution table further includes an information field, which may be used to store header information of the data newly written to the data table. When the historical data is moved, whether the head information of the historical data is consistent with the corresponding information field in the redistribution table or not can be judged, and when the head information of the historical data is consistent with the information field, the historical data is determined to be moved; when it is detected that the history data is not migrated, a new target node may be determined from the hash value calculated from the redistributed data of the data table and the target node calculation data, and the history data may be migrated to the new target node.
According to the read-write control method for the distributed database, the pre-built redistribution table is applied to the application scene of the expansion and contraction of the database, the read-write operation is carried out on the data table based on the state identification and the distribution key of the data table and the configuration number of new and old nodes of the database, the corresponding data table in the database is only required to be locked before the expansion and contraction of the database is carried out and in the process of modifying the redistribution table, and the data table is not required to be locked when the expansion and contraction of the database is carried out, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
Example IV
Fig. 5 is a schematic structural diagram of an embodiment of a read-write control device for a distributed database according to the present invention, which may be used to execute the method steps shown in fig. 2. As shown in fig. 5, the read-write control device of the distributed database may include: a preprocessing module 51, a read-write control module 52 and a redistribution table creation module 53.
Wherein, the redistribution table establishing module 53 is configured to establish a corresponding redistribution table according to the data table; the preprocessing module 51 is configured to modify a redistribution table according to a data table to be redistributed, where when modification of a content corresponding to the data table in the redistribution table is started, the data table is locked, and when modification of the content corresponding to the data table in the redistribution table is completed, the data table is unlocked, the redistribution table at least includes a state identifier of the data table, redistribution data and target node determining data, the state identifier is used to mark whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used to determine a target node storing the data table; the read-write control module 52 is configured to perform read-write operations based on the modified redistribution table according to the read-write instruction for the data table.
In the embodiment of the invention, a redistribution table can be constructed in advance for the distributed database. When a user wants to perform data redistribution operations such as expanding and shrinking the database and modifying the distribution key, the preprocessing module 51 may modify the pre-constructed redistribution table according to the data table to be redistributed in the database before performing data redistribution. For example, the state identification of the data table, the old and new distribution keys, the number of old and new node configurations of the database, etc. may be modified. Meanwhile, in modifying the redistribution table, it is necessary to lock the data table corresponding to the modified content, and during the locking, the data table cannot be read and written. The locks on the tables have been released during the data redistribution of the database. Therefore, if a read/write command is received for a certain data table in the database, the read/write control module 52 may perform a read/write operation on the data table based on the modified redistribution table.
According to the read-write control device of the distributed database, the read-write control of the data table is performed through the pre-built redistribution table, the corresponding data table in the database is only required to be locked before the data is redistributed in the database and in the process of modifying the redistribution table, and the data table is not required to be locked when the data is redistributed in the database, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
Example five
Fig. 6 is a schematic structural diagram of another embodiment of a read-write control device for a distributed database according to the present invention, which may be used to perform the method steps shown in fig. 3 or fig. 4. As shown in fig. 6, on the basis of the embodiment shown in fig. 5, the read-write control device of the distributed database provided by the embodiment of the invention can be applied to an application scenario of modifying a distribution key. Wherein the redistribution data is a new distribution key and an old distribution key of the data table, the target node determines the number of node configurations of the data for the database (i.e. the number of nodes allocated to the database), and the read-write control module 52 may include: a first control unit 521. The first control unit 521 may be configured to perform a read-write operation on the data table according to a read-write instruction for the data table based on a state identifier of the data table, a new distribution key and an old distribution key of the data table, and a node configuration number, where the new distribution key is a distribution key after modification of the data table, and the old distribution key is a distribution key before modification of the data table.
Specifically, when the received read-write instruction is an insert operation instruction, the first control unit 521 may be specifically configured to determine a state identifier of the data table; under the condition that the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a new distribution key and the node configuration number, and writing the inserted data into the new target node; and under the condition that the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the old distributing key and the node configuration number, and writing the inserted data into the old target node.
When the received read-write instruction is a query operation instruction, an update operation instruction, or a delete operation instruction, the first control unit 521 may be specifically configured to determine a status identifier of the data table; under the condition that the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a new distribution key and the node configuration number, determining an old target node according to a hash value calculated by a key value corresponding to an old distribution key and the node configuration number, and respectively performing data query operation, data update operation or data deletion operation in the new target node and the old target node; and under the condition that the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the old distributing key and the node configuration number, and performing data query operation, data updating operation or data deleting operation in the old target node.
In addition, the read-write control device of the distributed database provided by the embodiment of the invention can be also applied to the application scene of database expansion and contraction. Wherein, the redistribution data is a new node configuration number and an old node configuration number of the database, the target node determines that the data is a distribution key of the data table, and the read-write control module 52 may further include: a second control unit 522. The second control unit 522 may be configured to perform a read-write operation on the data table according to a read-write instruction for the data table, based on the state identifier and the distribution key of the data table, and the new node configuration number and the old node configuration number of the database, where the new node configuration number is the number of nodes of the database after the expansion and the old node configuration number is the number of nodes of the database before the expansion and the shrinkage.
Specifically, when the received read-write instruction is an insert operation instruction, the second control unit 522 may be specifically configured to determine a state identifier of the data table; under the condition that the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a distribution key and the configuration number of the new node, and writing the inserted data into the new target node; and under the condition that the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the distributing key and the configuration number of the old node, and writing the inserted data into the old target node.
When the received read-write instruction is a query operation instruction, an update operation instruction, or a delete operation instruction, the second control unit 522 may be specifically configured to determine a state identifier of the data table; under the condition that the data table is in a redistribution state, determining a new target node according to a hash value calculated by a key value corresponding to a distribution key and the new node configuration number, determining an old target node according to a hash value calculated by a key value corresponding to a distribution key and the old node configuration number, and respectively performing data query operation, data update operation or data deletion operation in the new target node and the old target node; and under the condition that the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the distributing key and the configuration number of the old node, and performing data query operation, data updating operation or data deleting operation in the old target node.
Furthermore, the read-write control device of the distributed database provided by the embodiment of the invention can be also applied to application scenes of moving the historical data. Specifically, the redistribution table further includes an information field, which may be used to store header information of the data newly written to the data table. The read-write control device of the distributed database may further include: a history data relocation module 61. The history data relocation module 61 may be configured to determine whether header information of the history data is consistent with corresponding information fields in the redistribution table when the history data is relocated, and determine that the history data is relocated when the header information of the history data is consistent with the information fields; and under the condition that the historical data is not moved, determining a new target node according to the hash value calculated by the redistribution data of the data table and the target node calculation data, and moving the historical data to the new target node.
According to the read-write control device for the distributed database, the pre-built redistribution table is applied to the application scene of modifying the distribution key or the expansion and contraction volume of the database, the read-write operation is carried out on the data table based on the modified redistribution table, the corresponding data table in the database is only required to be locked before the data table is subjected to the modification of the distribution key or the expansion and contraction volume of the database, and the data table is not required to be locked when the data table is subjected to the modification of the distribution key or the expansion and contraction volume of the database, so that the time of data blocking can be greatly shortened, the performance of the database is improved, and the influence of the data blocking on related services is reduced.
Example six
The internal functions and structures of a read-write control device of a distributed database are described above, which device may be implemented as an electronic device. Fig. 7 is a schematic structural diagram of an embodiment of an electronic device according to the present invention. As shown in fig. 7, the electronic device includes a memory 71 and a processor 72.
A memory 71 for storing a program. In addition to the programs described above, the memory 71 may also be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and the like.
The memory 71 may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
A processor 72 coupled to the memory 71, executing a program stored in the memory 71 for:
establishing a corresponding redistribution table according to a data table, wherein the redistribution table at least comprises a state identifier of the data table, redistribution data and target node determining data, the state identifier is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table;
Modifying the redistribution table according to the data table to be redistributed, wherein the data table is locked in the process of modifying the content corresponding to the data table in the redistribution table;
and performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table.
Further, as shown in fig. 7, the electronic device may further include: communication component 73, power component 74, audio component 75, display 76, and the like. Only some of the components are schematically shown in fig. 7, which does not mean that the electronic device only comprises the components shown in fig. 7.
The communication component 73 is configured to facilitate communication between the electronic device and other devices, either wired or wireless. The electronic device may access a wireless network based on a communication standard, such as WiFi,2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 73 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 73 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
A power supply assembly 74 provides power to the various components of the electronic device. The power components 74 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for electronic devices.
The audio component 75 is configured to output and/or input audio signals. For example, the audio component 75 includes a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 71 or transmitted via the communication component 73. In some embodiments, the audio component 75 further comprises a speaker for outputting audio signals.
The display 76 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.
Claims (17)
1. A read-write control method of a distributed database is characterized by comprising the following steps:
establishing a corresponding redistribution table according to a data table, wherein the redistribution table at least comprises a state identifier of the data table, redistribution data and target node determining data, the state identifier is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table;
Modifying the redistribution table according to a data table to be redistributed, wherein the data table is locked when the modification of the content corresponding to the data table in the redistribution table is started;
when the modification of the content corresponding to the data table in the redistribution table is completed, unlocking the data table;
and performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table.
2. The method according to claim 1, wherein the redistributed data is a new distribution key and an old distribution key of the data table, the target node determines the number of node configurations of the data for the data table, and the performing the read-write operation based on the modified redistributed table according to the read-write instruction for the data table includes:
and according to a read-write instruction aiming at the data table, performing read-write operation on the data table based on the state identification of the data table, a new distribution key, an old distribution key and the node configuration number, wherein the new distribution key is a distribution key after the data table is modified, and the old distribution key is a distribution key before the data table is modified.
3. The method according to claim 2, wherein the read-write command is an insert operation command, and the performing, according to the read-write command for the data table, the read-write operation on the data table based on the state identifier, the new distribution key, the old distribution key, and the node configuration number of the data table includes:
judging the state identification of the data table;
when the data table is in a redistribution state, determining a new target node according to a key value corresponding to the new distribution key and a hash value calculated by the node configuration number, and writing the inserted data into the new target node;
when the data table is in a non-redistributing state, determining an old target node according to the key value corresponding to the old distributing key and the hash value calculated by the node configuration number, and writing the inserted data into the old target node.
4. The method according to claim 2, wherein the read-write command is a query command, an update command, or a delete command, and the performing, according to the read-write command for the data table, the read-write operation on the data table based on the state identifier, the new distribution key, the old distribution key, and the node configuration number of the data table includes:
Judging the state identification of the data table;
when the data table is in a redistribution state, determining a new target node according to a key value corresponding to the new distribution key and a hash value calculated by the node configuration number, determining an old target node according to a key value corresponding to the old distribution key and a hash value calculated by the node configuration number, and respectively performing data query operation, data update operation or data deletion operation in the new target node and the old target node;
when the data table is in a non-redistributing state, determining an old target node according to the key value corresponding to the old distributing key and the hash value calculated by the node configuration number, and performing data query operation, data update operation or data deletion operation in the old target node.
5. The method according to claim 1, wherein the redistributing data is a new node configuration number and an old node configuration number of the database, the target node determines that the data is a distribution key of the data table, and the performing the read-write operation based on the modified redistributing table according to the read-write instruction for the data table includes:
And according to a read-write instruction aiming at the data table, performing read-write operation on the data table based on the state identification and the distribution key of the data table and the new node configuration number and the old node configuration number of the database, wherein the new node configuration number is the node number of the database after capacity expansion and contraction, and the old node configuration number is the node number of the database before capacity expansion and contraction.
6. The method according to claim 5, wherein the read-write command is an insert operation command, and the performing, according to the read-write command for the data table, the read-write operation on the data table based on the state identifier and the distribution key of the data table, and the new node configuration number and the old node configuration number of the database includes:
judging the state identification of the data table;
when the data table is in a redistribution state, determining a new target node according to a key value corresponding to the distribution key and a hash value calculated by the new node configuration number, and writing the inserted data into the new target node;
when the data table is in a non-redistributing state, determining an old target node according to the hash value calculated by the key value corresponding to the distributing key and the old node configuration number, and writing the inserted data into the old target node.
7. The method according to claim 5, wherein the read-write command is a query command, an update command, or a delete command, and the performing, according to the read-write command for the data table, the read-write operation on the data table based on the state identifier and the distribution key of the data table, and the new node configuration number and the old node configuration number of the database includes:
judging the state identification of the data table;
when the data table is in a redistribution state, determining a new target node according to the key value corresponding to the distribution key and the hash value calculated by the new node configuration number, determining an old target node according to the key value corresponding to the distribution key and the hash value calculated by the old node configuration number, and respectively performing data query operation, data update operation or data deletion operation in the new target node and the old target node;
when the data table is in a non-redistributing state, determining an old target node according to a key value corresponding to the distributing key and a hash value calculated by the old node configuration number, and performing data query operation, data update operation or data deletion operation in the old target node.
8. The method according to any one of claims 1 to 7, wherein the redistribution table further includes an information field for storing header information of data newly written to the data table, the method further comprising:
when the historical data is moved, judging whether the head information of the historical data is consistent with the information field in the redistribution table, and when the head information of the historical data is consistent with the information field, determining that the historical data is moved;
and when the history data is not moved, determining a new target node according to the hash value calculated by the redistribution data of the data table and the target node calculation data, and moving the history data to the new target node.
9. A read-write control device for a distributed database, comprising:
the system comprises a redistribution table establishing module, a redistribution table determining module and a data processing module, wherein the redistribution table establishing module is used for establishing a corresponding redistribution table according to a data table, the redistribution table at least comprises a state identifier of the data table, redistribution data and target node determining data, the state identifier is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table;
The preprocessing module is used for modifying the redistribution table according to the data table to be redistributed, wherein the data table is locked when the content corresponding to the data table in the redistribution table is modified, and the data table is unlocked when the content corresponding to the data table in the redistribution table is modified;
and the read-write control module is used for performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table.
10. The apparatus according to claim 9, wherein the redistributed data is a new distribution key and an old distribution key of the data table, the target node determines the number of node configurations of the data for the database, and the read-write control module includes:
the first control unit is used for performing read-write operation on the data table based on the state identification of the data table, a new distribution key, an old distribution key and the node configuration number according to the read-write instruction for the data table, wherein the new distribution key is a distribution key after modification of the data table, and the old distribution key is a distribution key before modification of the data table.
11. The device according to claim 10, wherein the read-write command is an insert operation command, the first control unit is specifically configured to,
judging the state identification of the data table;
under the condition that the data table is in a redistribution state, determining a new target node according to a key value corresponding to the new distribution key and a hash value calculated by the node configuration number, and writing the inserted data into the new target node;
and under the condition that the data table is in a non-redistributing state, determining an old target node according to the key value corresponding to the old distributing key and the hash value calculated by the node configuration number, and writing the inserted data into the old target node.
12. The device according to claim 10, wherein the read-write command is a query command, an update command or a delete command, and the first control unit is specifically configured to,
judging the state identification of the data table;
under the condition that the data table is in a redistribution state, determining a new target node according to a key value corresponding to the new distribution key and a hash value calculated by the node configuration number, determining an old target node according to a key value corresponding to the old distribution key and a hash value calculated by the node configuration number, and respectively performing data query operation, data update operation or data deletion operation in the new target node and the old target node;
And under the condition that the data table is in a non-redistributing state, determining an old target node according to the key value corresponding to the old distributing key and the hash value calculated by the node configuration number, and performing data query operation, data updating operation or data deleting operation in the old target node.
13. The apparatus according to claim 9, wherein the redistribution data is a new node configuration number and an old node configuration number of the database, the target node determines that the data is a distribution key of the data table, and the read-write control module includes:
and the second control unit is used for performing read-write operation on the data table based on the state identification and the distribution key of the data table and the new node configuration number and the old node configuration number of the database according to the read-write instruction for the data table, wherein the new node configuration number is the number of nodes of the database after capacity expansion and contraction, and the old node configuration number is the number of nodes of the database before capacity expansion and contraction.
14. The device according to claim 13, wherein the read-write command is an insert operation command, the second control unit is specifically configured to,
Judging the state identification of the data table;
under the condition that the data table is in a redistribution state, determining a new target node according to a key value corresponding to the distribution key and a hash value calculated by the new node configuration number, and writing the inserted data into the new target node;
and under the condition that the data table is in a non-redistributing state, determining an old target node according to the key value corresponding to the distributing key and the hash value calculated by the old node configuration number, and writing the inserted data into the old target node.
15. The device according to claim 13, wherein the read-write command is a query command, an update command or a delete command, and the second control unit is specifically configured to,
judging the state identification of the data table;
under the condition that the data table is in a redistribution state, determining a new target node according to the key value corresponding to the distribution key and the hash value calculated by the new node configuration number, determining an old target node according to the key value corresponding to the distribution key and the hash value calculated by the old node configuration number, and respectively performing data query operation, data update operation or data deletion operation in the new target node and the old target node;
And under the condition that the data table is in a non-redistributing state, determining an old target node according to the key value corresponding to the distributing key and the hash value calculated by the old node configuration number, and performing data query operation, data updating operation or data deleting operation in the old target node.
16. A read-write control device of a distributed database according to any of claims 9 to 15, wherein the redistribution table further comprises an information field for storing header information of data newly written to the data table, the device further comprising:
the historical data relocation module is used for judging whether the head information of the historical data is consistent with the information field in the redistribution table when the historical data is relocated, and determining that the historical data is relocated when the head information of the historical data is consistent with the information field; and under the condition that the historical data is not moved, determining a new target node according to the hash value calculated by the redistribution data of the data table and the target node calculation data, and moving the historical data to the new target node.
17. An electronic device, comprising:
a memory for storing a program;
a processor for running the program stored in the memory for:
establishing a corresponding redistribution table according to a data table, wherein the redistribution table at least comprises a state identifier of the data table, redistribution data and target node determining data, the state identifier is used for marking whether the data table is in a redistribution state, the redistribution data is change data of the data table before and after redistribution, and the target node determining data is used for determining a target node for storing the data table;
modifying the redistribution table according to a data table to be redistributed, wherein the data table is locked when the modification of the content corresponding to the data table in the redistribution table is started;
when the modification of the content corresponding to the data table in the redistribution table is completed, unlocking the data table;
and performing read-write operation based on the modified redistribution table according to the read-write instruction aiming at the data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811496200.5A CN111291112B (en) | 2018-12-07 | 2018-12-07 | Read-write control method and device for distributed database and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811496200.5A CN111291112B (en) | 2018-12-07 | 2018-12-07 | Read-write control method and device for distributed database and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111291112A CN111291112A (en) | 2020-06-16 |
| CN111291112B true CN111291112B (en) | 2023-04-28 |
Family
ID=71025249
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811496200.5A Active CN111291112B (en) | 2018-12-07 | 2018-12-07 | Read-write control method and device for distributed database and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111291112B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115146B (en) * | 2020-09-15 | 2023-09-15 | 北京人大金仓信息技术股份有限公司 | Data redistribution method, device, equipment and storage medium of database |
| CN117076465B (en) * | 2023-10-16 | 2024-04-05 | 支付宝(杭州)信息技术有限公司 | Data association query method and related equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521297A (en) * | 2011-11-30 | 2012-06-27 | 北京人大金仓信息技术股份有限公司 | Method for achieving system dynamic expansion in shared-nothing database cluster |
| CN105354315A (en) * | 2015-11-11 | 2016-02-24 | 华为技术有限公司 | Region division method in distributed database, Region node and system |
| US9805071B1 (en) * | 2016-11-10 | 2017-10-31 | Palantir Technologies Inc. | System and methods for live data migration |
| CN107305554A (en) * | 2016-04-20 | 2017-10-31 | 泰康保险集团股份有限公司 | Data query processing method and processing device |
| CN107357883A (en) * | 2017-06-30 | 2017-11-17 | 北京奇虎科技有限公司 | Data migration method and device |
| CN107547606A (en) * | 2016-06-29 | 2018-01-05 | 中兴通讯股份有限公司 | Data processing method, cluster manager dual system, explorer, data handling system |
| CN108319623A (en) * | 2017-01-18 | 2018-07-24 | 华为技术有限公司 | A kind of fast resampling method, apparatus and data-base cluster |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9830346B2 (en) * | 2013-11-26 | 2017-11-28 | Sap Se | Table redistribution in distributed databases |
-
2018
- 2018-12-07 CN CN201811496200.5A patent/CN111291112B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521297A (en) * | 2011-11-30 | 2012-06-27 | 北京人大金仓信息技术股份有限公司 | Method for achieving system dynamic expansion in shared-nothing database cluster |
| CN105354315A (en) * | 2015-11-11 | 2016-02-24 | 华为技术有限公司 | Region division method in distributed database, Region node and system |
| CN107305554A (en) * | 2016-04-20 | 2017-10-31 | 泰康保险集团股份有限公司 | Data query processing method and processing device |
| CN107547606A (en) * | 2016-06-29 | 2018-01-05 | 中兴通讯股份有限公司 | Data processing method, cluster manager dual system, explorer, data handling system |
| US9805071B1 (en) * | 2016-11-10 | 2017-10-31 | Palantir Technologies Inc. | System and methods for live data migration |
| CN108319623A (en) * | 2017-01-18 | 2018-07-24 | 华为技术有限公司 | A kind of fast resampling method, apparatus and data-base cluster |
| CN107357883A (en) * | 2017-06-30 | 2017-11-17 | 北京奇虎科技有限公司 | Data migration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111291112A (en) | 2020-06-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6348989B2 (en) | Firmware restoration method, apparatus, terminal, program, and recording medium | |
| US8819403B2 (en) | Method and device for performing data synchronization of a snapshot image by selectively reloading data from nonvolatile to volatile memory after wakeup from hibernation | |
| CN105718484A (en) | File writing method, file reading method, file deletion method, file query method and client | |
| JP5161119B2 (en) | File system idempotent journal mechanism | |
| CN108572789B (en) | Disk storage method and device, message pushing method and device and electronic equipment | |
| CN110874383B (en) | Data processing method and device and electronic equipment | |
| CN111291112B (en) | Read-write control method and device for distributed database and electronic equipment | |
| US10838944B2 (en) | System and method for maintaining a multi-level data structure | |
| CN110874358B (en) | Multi-attribute column storage and retrieval method and device and electronic equipment | |
| US11809813B1 (en) | Multilayered electronic content management system | |
| CN105678183A (en) | User data management method and device for intelligent terminal | |
| CN106021027A (en) | Terminal data processing method and system | |
| US9456412B2 (en) | Electronic device and method for selecting access point name | |
| KR20150045073A (en) | Data Operating Method And System supporting the same | |
| EP3343395A1 (en) | Data storage method and apparatus for mobile terminal | |
| CN110647421B (en) | Database processing method, device and system and electronic equipment | |
| CN110555075B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN114928620B (en) | User information synchronization method, apparatus, device, storage medium, and program product | |
| CN111831472B (en) | Snapshot creation method and device and electronic equipment | |
| CN109213431A (en) | The consistency detecting method and device and electronic equipment of more copy datas | |
| CN110795318B (en) | Data processing method and device and electronic equipment | |
| CN112988860B (en) | Data acceleration processing method and device and electronic equipment | |
| CN110968557B (en) | Data processing method and device in distributed file system and electronic equipment | |
| CN110866003B (en) | Index value number estimation method and device and electronic equipment | |
| CN102930619B (en) | Large-capacity access control system and control method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |