CN111522811B - Database processing method and device, storage medium and terminal - Google Patents
Database processing method and device, storage medium and terminal Download PDFInfo
- Publication number
- CN111522811B CN111522811B CN202010190447.5A CN202010190447A CN111522811B CN 111522811 B CN111522811 B CN 111522811B CN 202010190447 A CN202010190447 A CN 202010190447A CN 111522811 B CN111522811 B CN 111522811B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- data node
- storage
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database processing method and device, a storage medium and a terminal, relates to the technical field of data processing, and mainly aims to solve the problem of low data processing efficiency during capacity expansion of an existing database. The method mainly comprises the following steps: determining a first data storage corresponding relation of an original data node by using the number of the original data node and a preset virtual node number, and determining a second data storage corresponding relation of a current data node by using the number of the current data node and the preset virtual node number; determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation; and storing the target data in the original data node into the corresponding current data node according to the mapping relation.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a database processing method and apparatus, a storage medium, and a terminal.
Background
With the advent of the big data era, the data storage scale is gradually increased, and especially for the processing of a single data node in a relational database, the pressure is extremely large.
At present, the existing database is used for targeting data in a form of table division, the database is divided into a plurality of small tables in a form of a large table, the data are distributed and stored in different tables according to a specific algorithm, in the process of expanding the database, increasing the number of nodes according to modular operation can cause that all data in the database are required to be redistributed and stored every time when the nodes are newly increased, and increasing the number of nodes which is even times of the original number of nodes can cause more than half of data migration, which causes unnecessary data resource waste, in addition, the newly increased data nodes can generate a disturbance problem of data jitter when simultaneously performing reading and writing operations, and the newly written data can be unevenly distributed when being stored in the expanded database, which causes that the efficiency of the database for processing the data is reduced, thereby affecting the performance of the database.
Disclosure of Invention
In view of this, the present invention provides a method and an apparatus for processing a database, a storage medium, and a terminal, and mainly aims to solve the problem of low data processing efficiency when an existing database is expanded.
According to an aspect of the present invention, there is provided a database processing method, including:
determining a first data storage corresponding relation of an original data node by using the number of the original data node and a preset virtual node number, and determining a second data storage corresponding relation of a current data node by using the number of the current data node and the preset virtual node number;
determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation;
and storing the target data in the original data node into the corresponding current data node according to the mapping relation.
Further, the determining a first data storage corresponding relationship of the original data node by using the original data node number and the preset virtual node number, and determining a second data storage corresponding relationship of the current data node by using the current data node number and the preset virtual node number includes:
calculating a first hash value of an original data node through the number of the original data node and a preset virtual node number, dividing a first storage interval according to the first hash value, and establishing a first data storage corresponding relation by using the hash values of the first storage interval and at least one target data;
calculating a second hash value of the original data node according to the number of the current data nodes and the number of preset virtual nodes, dividing a second storage interval according to the second hash value, and establishing a second data storage corresponding relation by using the hash values of the second storage interval and at least one target data.
Further, the preset virtual node number is used for representing hash value intervals of the division corresponding relation in the first data storage corresponding relation and the second data storage corresponding relation.
Further, the determining, according to the first data storage corresponding relationship and the second data storage corresponding relationship, a mapping relationship of the original data node to the current data node for data storage includes:
and matching the first data storage corresponding relation with the second data storage corresponding relation according to the first storage interval and the second storage interval, and establishing a mapping relation of the original data node for data storage to the current data node according to a matching result.
Further, after the target data in the original data node is stored into the corresponding current data node according to the mapping relationship, the method further includes:
checking whether the target data in the current data node is completely stored;
and if so, deleting the target data in the original data node.
Further, the checking whether the target data in the current data node is completely stored includes:
and inquiring whether the target data in the current data node is matched with the target data in the original data node according to the data number and the data type.
Further, the method further comprises:
and updating the preset virtual node number according to the original data node number and the current data node number.
According to another aspect of the present invention, there is provided a database processing apparatus, including:
the first determining module is used for determining a first data storage corresponding relation of an original data node by using the number of the original data node and a preset virtual node number, and determining a second data storage corresponding relation of a current data node by using the number of the current data node and the preset virtual node number;
a second determining module, configured to determine, according to the first data storage corresponding relationship and the second data storage corresponding relationship, a mapping relationship of the original data node to the current data node for data storage;
and the storage module is used for storing the target data in the original data node into the corresponding current data node according to the mapping relation.
Further, the first determining module comprises:
the first determining unit is used for calculating a first hash value of an original data node through the number of the original data node and the number of preset virtual nodes, dividing a first storage interval according to the first hash value, and establishing a first data storage corresponding relation by using the hash value of the first storage interval and at least one piece of target data;
and the second determining unit is used for calculating a second hash value of the original data node according to the current data node number and the preset virtual node number, dividing a second storage interval according to the second hash value, and establishing a second data storage corresponding relation by using the hash values of the second storage interval and at least one target data.
Further, the preset number of virtual nodes is used to represent hash value intervals of the first data storage corresponding relationship and the second data storage corresponding relationship, which are divided into corresponding relationships.
Further, the second determining module is specifically configured to match the first data storage corresponding relationship with the second data storage corresponding relationship according to the first storage interval and the second storage interval, and establish a mapping relationship of the original data node to perform data storage on the current data node according to a matching result.
Further, the apparatus further comprises:
the checking module is used for checking whether the target data in the current data node is completely stored;
and the deleting module is used for deleting the target data in the original data node if the target data in the original data node is the original data node.
Further, the check module is specifically configured to query whether the target data in the current data node is matched with the target data in the original data node according to the number of data and the type of data.
Further, the apparatus further comprises:
and the updating module is used for updating the preset virtual node number according to the original data node number and the current data node number.
According to another aspect of the present invention, there is provided a storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the processing method of the database.
According to still another aspect of the present invention, there is provided a terminal including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the processing method of the database.
By the technical scheme, the technical scheme provided by the embodiment of the invention at least has the following advantages:
compared with the prior art that increasing the number of nodes according to modular operation causes that all data in the database needs to be redistributed and stored every time nodes are newly increased, and that increasing node data with even times of the original number of nodes causes more than half of data migration, thereby causing unnecessary data resource waste, the embodiment of the invention determines the first data storage corresponding relation of the original data nodes by using the original data nodes and the preset virtual nodes, and determines the second data storage corresponding relation of the current data nodes by using the current data nodes and the preset virtual nodes; determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation; and storing the target data in the original data nodes into the corresponding current data nodes according to the mapping relation, so as to realize smooth transition of the data in the capacity expansion process of the database, stably store the data in the original data nodes before capacity expansion into the corresponding current data nodes after capacity expansion, reduce migration volume of data migration, avoid uneven data storage in the capacity expansion process of the database, reduce data fluctuation and improve the processing efficiency of the data of the database.
The above description is only an overview of the technical solutions of the present invention, and the present invention can be implemented in accordance with the content of the description so as to make the technical means of the present invention more clearly understood, and the above and other objects, features, and advantages of the present invention will be more clearly understood.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flow chart of a database processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another database processing method provided by the embodiment of the invention;
FIG. 3 is a block diagram illustrating a processing apparatus of a database according to an embodiment of the present invention;
FIG. 4 is a block diagram of another database processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
An embodiment of the present invention provides a database processing method, as shown in fig. 1, the method includes:
101. and determining a first data storage corresponding relation of the original data nodes by using the number of the original data nodes and the number of preset virtual nodes, and determining a second data storage corresponding relation of the current data nodes by using the number of the current data nodes and the number of the preset virtual nodes.
The original data nodes are data storage nodes of nodes not added in the database, the current data nodes are data storage nodes of nodes added in the database, correspondingly, the number of the original data nodes is the total number of the original data nodes contained in the database, and the number of the current data nodes is the total number of the current data nodes contained in the database. The preset virtual node number is used for representing hash value intervals of the first data storage corresponding relation and the second data storage corresponding relation, so that the hash value range of the original data node can be calculated through the original data node number and the preset virtual node number, and the first data storage corresponding relation is determined according to the relation between the hash value corresponding to the data in each node of the original data node and the hash value range. Similarly, the hash value range of the current data node can be calculated through the number of the current data nodes and the number of the preset virtual nodes, and the corresponding relation of the second data storage is determined according to the relation between the hash value corresponding to the data in each node in the current data node and the hash value range.
102. And determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation.
Since the first data storage corresponding relationship includes a corresponding relationship of stored data of an original data node, and the second data storage corresponding relationship includes a corresponding relationship of stored data of a current data node, a corresponding mapping relationship of data storage from the original data node to the current data node is determined according to the first data storage corresponding relationship and the second data storage corresponding relationship, for example, data a is stored in the original data node 2 according to the first data storage corresponding relationship, data a is stored in the current data node 3 according to the second data storage corresponding relationship, and after comparison, data a is stored from the original data node 2 to the current data node 3.
103. And storing the target data in the original data node into the corresponding current data node according to the mapping relation.
In the embodiment of the invention, in order to ensure the stability of the data storage process, the target data in the original data node is stored to the corresponding position in the current data node according to the uniquely determined mapping relation.
It should be noted that, both the original data node and the current data node may be at least one data storage node for storing a plurality of data, so that when the storage is performed according to the mapping relationship, the target data in the original data node may be accurately and stably migrated to the current data node, thereby completing the capacity expansion of the database.
Compared with the prior art that increasing the number of nodes according to modular operation causes that all data in the database needs to be redistributed and stored every time when the number of nodes is increased, and that increasing node data with even times of the number of original nodes causes more than half of data migration, thereby causing unnecessary data resource waste, the embodiment of the invention determines the first data storage corresponding relation of the original data nodes by using the number of the original data nodes and the number of the preset virtual nodes, and determines the second data storage corresponding relation of the current data nodes by using the number of the current data nodes and the number of the preset virtual nodes; determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation; and storing the target data in the original data nodes into the corresponding current data nodes according to the mapping relation, so as to realize smooth transition of the data in the capacity expansion process of the database, and stably storing the data in the original data nodes before capacity expansion into the corresponding current data nodes after capacity expansion, so that the uneven data storage in the capacity expansion process of the database is avoided, the data fluctuation is reduced, and the processing efficiency of the data in the database is improved.
An embodiment of the present invention provides another database processing method, as shown in fig. 2, the method includes:
201. calculating a first hash value of an original data node through the number of the original data node and a preset virtual node number, dividing a first storage interval according to the first hash value, and establishing a first data storage corresponding relation by using the hash values of the first storage interval and at least one target data.
For the embodiment of the invention, in order to accurately divide node positions of different data storage, a first hash value of an original data node is calculated by using the number of the original data node and the number of preset virtual nodes, wherein the number of the preset virtual nodes is used as the number of the first hash value, and the first hash value can be calculated by an expression long h = rsHash (n + Math.pow (2, i)), wherein n is the number of the original data node, i is a cycle value of the number of the preset virtual nodes, and i =1,2,3, k, k is the number of the preset virtual nodes. For example, the original data node is 3, the number of preset virtual nodes is 100, and the original nodes stored correspondingly to different hash values are circularly divided according to the calculation expression of the hash values long h = rsHash (3 + math (2, i)), so that 100 hash values can be calculated, for example, data corresponding to hash values of 0 to 10 are stored in the node 1, data corresponding to hash values of 10 to 20 are stored in the node 2, data corresponding to hash values of 20 to 30 are stored in the node 3, data corresponding to hash values of 30 to 40 are stored in the node 1, data corresponding to hash values of 40 to 50 are stored in the node 2, and so on, until all data corresponding to hash values are stored in 3 nodes according to circular distribution, that is, the first storage interval is divided according to the first hash value. In addition, a first data storage corresponding relationship is established through the hash values of the first storage interval and the at least one target data, that is, a corresponding relationship is established according to hash value ranges corresponding to different nodes in the first storage interval and the hash values of the target data, so that the stored nodes can be quickly located through the hash values of one data. For example, the hash value of the data a is 22, and the node storage interval 22 divided according to the hash value interval in the first storage interval is stored in the node 4, so that the first data correspondence relationship is established such that the data a is stored in the node 4.
It should be noted that, in the embodiment of the present invention, different data nodes may correspond to different distribution intervals of hash values, so that one node may store data corresponding to the distribution intervals of multiple hash values, that is, the first storage interval is a distribution interval including hash values corresponding to multiple nodes, which is not limited in the embodiment of the present invention.
Further, in the embodiment of the present invention, before step 201, the method further includes: and updating the preset virtual node number according to the original data node number and the current data node number.
For the embodiment of the present invention, since the preset virtual nodes are configured in advance, in order to make the number of the preset virtual nodes more conform to data migration between the original data node and the current data node and improve stable row and migration smoothness of data storage, the number of the preset virtual nodes is updated according to the number of the original data nodes and the current data node, specifically, an update list may be configured in advance, and a data difference between the number of the original data nodes and the current data node is searched from the list to update the number of the matched preset virtual nodes, for example, if the difference between the number of the original data nodes and the current data node is 100, the number of the preset virtual nodes is 500, which is not limited in the embodiment of the present invention.
202. Calculating a second hash value of the original data node according to the number of the current data nodes and the number of preset virtual nodes, dividing a second storage interval according to the second hash value, and establishing a second data storage corresponding relation by using the hash values of the second storage interval and at least one target data.
The implementation principle of this step is the same as that in step 201 shown in fig. 2, only the number of original data nodes is replaced with the number of current data nodes to calculate the second hash value of the current data node, a second storage interval is divided, and the second data storage corresponding relationship is established in combination with the hash value of the target data, which is not described herein again.
203. And matching the first data storage corresponding relation with the second data storage corresponding relation according to the first storage interval and the second storage interval, and establishing a mapping relation of the original data node for data storage to the current data node according to a matching result.
For the embodiment of the invention, in order to migrate the data in the original data node to the current data node in a uniform and stable manner, the first data storage corresponding relation and the second data storage corresponding relation are matched according to the first storage interval and the second storage interval, and a migration mapping relation is established. The matching of the first data storage corresponding relationship and the second data storage corresponding relationship is a matching conversion of the nodes of the data in the first storage interval and the second storage interval respectively corresponding to the nodes in the first data storage corresponding relationship and the second data storage corresponding relationship, for example, the first storage interval is a hash value of 20 to 50, the first storage interval is corresponding to the node 4 according to the first data storage corresponding relationship, the second storage interval is a hash value of 20 to 50, the second storage interval is corresponding to the node 6 according to the second data storage corresponding relationship, and the mapping relationship of the first data storage corresponding relationship and the second data storage corresponding relationship is the node 4 to the node 6.
204. And storing the target data in the original data node into the corresponding current data node according to the mapping relation.
This step is the same as step 103 shown in fig. 1, and is not described herein again.
205. And checking whether the target data in the current data node is completely stored.
For the embodiment of the present invention, in order to avoid that the data in the database is lost in the processing process, it is checked whether the target data in the current data node is completely stored, that is, whether the data stored from the original data node to the current data node is complete, for example, the original data node 1 includes data a, b, c, and d, and it is checked whether the original data node 2 includes data a, b, c, and d from the original data node 1, which is not specifically limited in the embodiment of the present invention.
For further explanation and limitation, step 205 in the embodiment of the present invention specifically includes: and inquiring whether the target data in the current data node is matched with the target data in the original data node or not according to the data number and the data type.
The data source stored in the current data node is the original data node, so that in order to accurately check the data, the target data in the current data node is queried according to the data number and the data type, and if the queried target data is the same as the data number and the data type of the target data in the original data node, the target data in the original data node is completely stored in the current data node.
206. And if so, deleting the target data in the original data node.
For the embodiment of the invention, in order to complete data migration of the capacity expansion database, when it is verified that the data of the original data node is completely stored in the current data node, the target data of the original data node is deleted, and release of redundant data is realized. In addition, the deleted target data is only the target data of the original data node corresponding to the current data node which passes the verification, and the storage of other data is not influenced.
Compared with the prior art that increasing the number of nodes according to modular operation causes that all data in the database needs to be redistributed and stored every time nodes are newly added, and that increasing node data with even times of the number of original nodes causes more than half of data migration, thereby causing unnecessary data resource waste, the embodiment of the invention determines the first data storage corresponding relation of the original data nodes by using the number of the original data nodes and the number of preset virtual nodes, and determines the second data storage corresponding relation of the current data nodes by using the number of the current data nodes and the number of the preset virtual nodes; determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation; and storing the target data in the original data nodes into the corresponding current data nodes according to the mapping relation, so as to realize smooth transition of the data in the capacity expansion process of the database, and stably storing the data in the original data nodes before capacity expansion into the corresponding current data nodes after capacity expansion, so that the uneven data storage in the capacity expansion process of the database is avoided, the data fluctuation is reduced, and the processing efficiency of the data in the database is improved.
Further, as an implementation of the method shown in fig. 1, an embodiment of the present invention provides a database processing apparatus, and as shown in fig. 3, the apparatus includes: a first determining module 31, a second determining module 32 and a storage module 33.
A first determining module 31, configured to determine a first data storage corresponding relationship of an original data node by using the number of original data nodes and the number of preset virtual nodes, and determine a second data storage corresponding relationship of a current data node by using the number of current data nodes and the number of preset virtual nodes;
a second determining module 32, configured to determine, according to the first data storage corresponding relationship and the second data storage corresponding relationship, a mapping relationship of the original data node to the current data node for data storage;
and the storage module 33 is configured to store the target data in the original data node into the corresponding current data node according to the mapping relationship.
Compared with the prior art that increasing the number of nodes according to modular operation causes that all data in the database needs to be redistributed and stored every time nodes are newly added, and that increasing node data of even times of the original number of nodes causes more than half of data migration, thereby causing unnecessary data resource waste, the embodiment of the invention determines the first data storage corresponding relation of the original data nodes by using the original data nodes and the preset virtual nodes, and determines the second data storage corresponding relation of the current data nodes by using the current data nodes and the preset virtual nodes; determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation; and storing the target data in the original data nodes into the corresponding current data nodes according to the mapping relation, so as to realize smooth transition of the data in the capacity expansion process of the database, and stably storing the data in the original data nodes before capacity expansion into the corresponding current data nodes after capacity expansion, so that the uneven data storage in the capacity expansion process of the database is avoided, the data fluctuation is reduced, and the processing efficiency of the data in the database is improved.
Further, as an implementation of the method shown in fig. 2, an embodiment of the present invention provides another database processing apparatus, and as shown in fig. 4, the apparatus includes: a first determining module 41, a second determining module 42, a storing module 43, a checking module 44, a deleting module 45, and an updating module 46.
A first determining module 41, configured to determine a first data storage corresponding relationship of an original data node by using the number of the original data node and a preset virtual node, and determine a second data storage corresponding relationship of a current data node by using the number of the current data node and the preset virtual node;
a second determining module 42, configured to determine, according to the first data storage corresponding relationship and the second data storage corresponding relationship, a mapping relationship of the original data node to the current data node for data storage;
and a storage module 43, configured to store the target data in the original data node into a corresponding current data node according to the mapping relationship.
Further, the first determining module 41 includes:
a first determining unit 4101, configured to calculate a first hash value of an original data node by using the number of the original data node and a preset number of virtual nodes, divide a first storage interval according to the first hash value, and establish a first data storage correspondence relationship by using the hash values of the first storage interval and at least one target data;
a second determining unit 4102, configured to calculate a second hash value of the original data node according to the current data node number and the preset virtual node number, divide a second storage interval according to the second hash value, and establish a second data storage corresponding relationship by using the hash values of the second storage interval and at least one target data.
Further, the preset number of virtual nodes is used to represent hash value intervals of the first data storage corresponding relationship and the second data storage corresponding relationship, which are divided into corresponding relationships.
Further, the second determining module 42 is specifically configured to match the first data storage corresponding relationship with the second data storage corresponding relationship according to the first storage interval and the second storage interval, and establish a mapping relationship between the original data node and the current data node for data storage according to a matching result.
Further, the apparatus further comprises:
a checking module 44, configured to check whether target data in the current data node is completely stored;
and a deleting module 45, configured to delete the target data in the original data node if the target data is the same as the original data node.
Further, the checking module 44 is specifically configured to query whether the target data in the current data node is matched with the target data in the original data node according to the number of data and the data type.
Further, the apparatus further comprises:
and an updating module 46, configured to update the preset number of virtual nodes according to the number of original data nodes and the number of current data nodes.
Compared with the prior art that increasing the number of nodes according to modular operation causes that all data in the database needs to be redistributed and stored every time nodes are newly added, and that increasing node data with even times of the number of original nodes causes more than half of data migration, thereby causing unnecessary data resource waste, the embodiment of the invention determines the first data storage corresponding relation of the original data nodes by using the number of the original data nodes and the number of preset virtual nodes, and determines the second data storage corresponding relation of the current data nodes by using the number of the current data nodes and the number of the preset virtual nodes; determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation; and storing the target data in the original data nodes into the corresponding current data nodes according to the mapping relation, so as to realize smooth transition of the data in the capacity expansion process of the database, and stably storing the data in the original data nodes before capacity expansion into the corresponding current data nodes after capacity expansion, so that the uneven data storage in the capacity expansion process of the database is avoided, the data fluctuation is reduced, and the processing efficiency of the data in the database is improved.
According to an embodiment of the present invention, a storage medium is provided, and the storage medium stores at least one executable instruction, and the computer executable instruction can execute the processing method of the database in any method embodiment.
Fig. 5 is a schematic structural diagram of a terminal according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the terminal.
As shown in fig. 5, the terminal may include: a processor (processor) 502, a Communications Interface 504, a memory 506, and a communication bus 508.
Wherein: the processor 502, communication interface 504, and memory 506 communicate with one another via a communication bus 508.
A communication interface 504 for communicating with network elements of other devices, such as clients or other servers.
The processor 502 is configured to execute the program 510, and may specifically execute relevant steps in the above-described database processing method embodiment.
In particular, program 510 may include program code that includes computer operating instructions.
The processor 502 may be a central processing unit CPU, or an Application Specific Integrated Circuit ASIC (Application Specific Integrated Circuit), or one or more Integrated circuits configured to implement an embodiment of the present invention. The terminal comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
A memory 506 for storing a program 510. The memory 506 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 510 may specifically be used to cause the processor 502 to perform the following operations:
determining a first data storage corresponding relation of an original data node by using the number of the original data node and a preset virtual node number, and determining a second data storage corresponding relation of a current data node by using the number of the current data node and the preset virtual node number;
determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation;
and storing the target data in the original data node into the corresponding current data node according to the mapping relation.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized in a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a memory device and executed by a computing device, and in some cases, the steps shown or described may be executed out of order, or separately as individual integrated circuit modules, or multiple modules or steps thereof may be implemented as a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A method for processing a database, comprising:
determining a first data storage corresponding relation of an original data node by using the number of the original data node and a preset virtual node number, and determining a second data storage corresponding relation of a current data node by using the number of the current data node and the preset virtual node number;
determining a mapping relation of the original data node to the current data node for data storage according to the first data storage corresponding relation and the second data storage corresponding relation;
storing the target data in the original data node into a corresponding current data node according to the mapping relation;
the determining of the first data storage corresponding relation of the original data node by using the original data node number and the preset virtual node number and the determining of the second data storage corresponding relation of the current data node by using the current data node number and the preset virtual node number comprise:
calculating a first hash value of an original data node through the number of the original data node and a preset virtual node number, dividing a first storage interval according to the first hash value, and establishing a first data storage corresponding relation by using the hash values of the first storage interval and at least one target data;
calculating a second hash value of the original data node according to the number of the current data nodes and the number of preset virtual nodes, dividing a second storage interval according to the second hash value, and establishing a second data storage corresponding relation by using the hash values of the second storage interval and at least one target data.
2. The method of claim 1, wherein the preset number of virtual nodes is used to represent hash value intervals of the first data storage correspondence and the second data storage correspondence, which divide the correspondence.
3. The method of claim 2, wherein determining the mapping relationship for the original data node to store data to the current data node according to the first data storage correspondence and the second data storage correspondence comprises:
and matching the first data storage corresponding relation with the second data storage corresponding relation according to the first storage interval and the second storage interval, and establishing a mapping relation of the original data node for data storage to the current data node according to a matching result.
4. The method of claim 3, wherein after storing the target data in the original data node into the corresponding current data node according to the mapping relationship, the method further comprises:
checking whether the target data in the current data node is completely stored;
and if so, deleting the target data in the original data node.
5. The method of claim 4, wherein the checking whether the target data in the current data node is completely stored comprises:
and inquiring whether the target data in the current data node is matched with the target data in the original data node or not according to the data number and the data type.
6. The method of claim 1, further comprising:
and updating the preset virtual node number according to the original data node number and the current data node number.
7. A database processing apparatus, comprising:
the first determining module is used for determining a first data storage corresponding relation of an original data node by using the number of the original data node and a preset virtual node number, and determining a second data storage corresponding relation of a current data node by using the number of the current data node and the preset virtual node number;
a second determining module, configured to determine, according to the first data storage corresponding relationship and the second data storage corresponding relationship, a mapping relationship of the original data node to the current data node for data storage;
the storage module is used for storing the target data in the original data node into the corresponding current data node according to the mapping relation;
wherein the first determining module comprises:
the first determining unit is used for calculating a first hash value of an original data node through the number of the original data node and the number of preset virtual nodes, dividing a first storage interval according to the first hash value, and establishing a first data storage corresponding relation by using the hash values of the first storage interval and at least one target data;
and the second determining unit is used for calculating a second hash value of the original data node according to the current data node number and the preset virtual node number, dividing a second storage interval according to the second hash value, and establishing a second data storage corresponding relation by using the hash values of the second storage interval and at least one target data.
8. A storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the database processing method according to any one of claims 1 to 6.
9. A terminal, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface are communicated with each other through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the corresponding operation of the processing method of the database according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010190447.5A CN111522811B (en) | 2020-03-18 | 2020-03-18 | Database processing method and device, storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010190447.5A CN111522811B (en) | 2020-03-18 | 2020-03-18 | Database processing method and device, storage medium and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111522811A CN111522811A (en) | 2020-08-11 |
| CN111522811B true CN111522811B (en) | 2023-04-07 |
Family
ID=71901884
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010190447.5A Active CN111522811B (en) | 2020-03-18 | 2020-03-18 | Database processing method and device, storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111522811B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112416264A (en) * | 2020-12-11 | 2021-02-26 | 中国建设银行股份有限公司 | Data storage method and device and computer storage medium |
| CN112698926B (en) * | 2021-03-25 | 2021-07-02 | 成都新希望金融信息有限公司 | Data processing method, device, equipment, storage medium and system |
| CN113794558B (en) * | 2021-09-16 | 2024-02-27 | 烽火通信科技股份有限公司 | L-tree calculation method, device and system in XMS algorithm |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009295127A (en) * | 2008-06-09 | 2009-12-17 | Nippon Telegr & Teleph Corp <Ntt> | Access method, access device and distributed data management system |
| CN102968503A (en) * | 2012-12-10 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Data processing method for database system, and database system |
| CN104298541A (en) * | 2014-10-22 | 2015-01-21 | 浪潮(北京)电子信息产业有限公司 | Data distribution algorithm and data distribution device for cloud storage system |
| CN108334551A (en) * | 2017-12-29 | 2018-07-27 | 谷米科技有限公司 | Date storage method and system, data query method and system |
| CN109254729A (en) * | 2018-08-24 | 2019-01-22 | 杭州宏杉科技股份有限公司 | A kind of method and apparatus of object storage |
-
2020
- 2020-03-18 CN CN202010190447.5A patent/CN111522811B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009295127A (en) * | 2008-06-09 | 2009-12-17 | Nippon Telegr & Teleph Corp <Ntt> | Access method, access device and distributed data management system |
| CN102968503A (en) * | 2012-12-10 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Data processing method for database system, and database system |
| CN104298541A (en) * | 2014-10-22 | 2015-01-21 | 浪潮(北京)电子信息产业有限公司 | Data distribution algorithm and data distribution device for cloud storage system |
| CN108334551A (en) * | 2017-12-29 | 2018-07-27 | 谷米科技有限公司 | Date storage method and system, data query method and system |
| CN109254729A (en) * | 2018-08-24 | 2019-01-22 | 杭州宏杉科技股份有限公司 | A kind of method and apparatus of object storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111522811A (en) | 2020-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111522811B (en) | Database processing method and device, storage medium and terminal | |
| US10628449B2 (en) | Method and apparatus for processing database data in distributed database system | |
| CN108287660B (en) | Data storage method and device | |
| CN110147407B (en) | Data processing method and device and database management server | |
| US20140222870A1 (en) | System, Method, Software, and Data Structure for Key-Value Mapping and Keys Sorting | |
| CN108874803A (en) | Date storage method, device and storage medium | |
| CN111722918A (en) | Service identification code generation method and device, storage medium and electronic equipment | |
| CN111723073B (en) | Data storage processing method, device, processing system and storage medium | |
| CN111917834A (en) | Data synchronization method and device, storage medium and computer equipment | |
| CN112148693A (en) | Data processing method, device and storage medium | |
| CN116578746A (en) | Object de-duplication method and device | |
| CN114791907A (en) | Method and system for processing multi-tenant shared data | |
| CN111046106A (en) | Cache data synchronization method, device, equipment and medium | |
| CN114218266A (en) | Data query method and device, electronic equipment and storage medium | |
| CN107656980B (en) | Method applied to distributed database system and distributed database system | |
| CN112579591B (en) | Data verification method, device, electronic equipment and computer readable storage medium | |
| CN108306976B (en) | SDN controller architecture used in cloud computing network virtualization | |
| CN116303343A (en) | Data slicing method, device, electronic equipment and storage medium | |
| JP6189266B2 (en) | Data processing apparatus, data processing method, and data processing program | |
| CN116578641A (en) | Database separation method and system based on ketama algorithm | |
| CN116303418A (en) | Mass rule processing method, system and storage medium based on Internet of things platform | |
| CN115480704A (en) | Method, device and equipment for constructing data block group migration mapping table and storage medium | |
| CN114048219A (en) | Graph database updating method and device | |
| CN109962942B (en) | Copy update method and apparatus, and computer-readable storage medium | |
| CN115129779A (en) | Database synchronization method, device and readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |