CN115858539A - Data storage method and device, electronic equipment and storage medium - Google Patents
Data storage method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115858539A CN115858539A CN202211708814.1A CN202211708814A CN115858539A CN 115858539 A CN115858539 A CN 115858539A CN 202211708814 A CN202211708814 A CN 202211708814A CN 115858539 A CN115858539 A CN 115858539A
- Authority
- CN
- China
- Prior art keywords
- node
- target
- data
- target node
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a data storage method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a first target node of the candidate nodes in response to the storage request; acquiring target node information of each second target node in the candidate nodes through the first target node, and calculating target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function; and determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, and storing the data table to be stored to the target storage node. By operating the technical scheme provided by the embodiment of the invention, the problems that the single server cannot support the requirements of storage and calculation along with the increase of the data amount and the increase of the number of data tables can be solved, and the storable amount of the data is expanded.
Description
Technical Field
The present invention relates to computer technologies, and in particular, to a data storage method and apparatus, an electronic device, and a storage medium.
Background
In the field of privacy computing and the like, data is stored in the form of data tables, and the number of data tables increases as the amount of data increases.
In the prior art, all data tables are usually stored directly in a single server, and as the amount of data to be stored increases, the single server may not be able to support the storage and computation requirements.
Disclosure of Invention
The invention provides a data storage method, a data storage device, electronic equipment and a storage medium, which are used for expanding the storable amount of data.
According to an aspect of the present invention, there is provided a data storage method, the method including:
determining a first target node of the candidate nodes in response to the storage request;
acquiring target node information of each second target node in the candidate nodes through the first target node, and calculating target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function;
and determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, and storing the data table to be stored to the target storage node.
According to another aspect of the present invention, there is provided a data storage device comprising:
a first node determination module to determine a first target node of the candidate nodes in response to the storage request;
the first weight calculation module is used for acquiring target node information of each second target node in the candidate nodes through the first target node and calculating the target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function;
and the first data table storage module is used for determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, and storing the data table to be stored to the target storage node.
According to another aspect of the present invention, there is provided a data storage method, the method including:
storing the data to be stored into a data table to be stored, and generating a data table identifier and the number of the data tables;
and creating a target directory associated with each data table to be stored in the current server according to the number of the data tables and a preset directory table number threshold, and storing the data tables to be stored in the target directory.
According to another aspect of the present invention, there is provided a data storage device comprising:
the data storage module is used for storing the data to be stored into a data table to be stored and generating data table identification and data table quantity;
and the second data table storage module is used for creating a target directory associated with each data table to be stored in the current server according to the number of the data tables and a preset directory table number threshold value, and storing the data table to be stored in the target directory.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data storage method of any of the embodiments of the invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a data storage method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme provided by the embodiment, the target node information of each second target node in the candidate nodes is obtained through the first target node in the candidate nodes, the target node weight of each second target node is obtained through calculation according to the target node information, the node capable of storing the data table to be stored in the second target node is determined according to the target node weight, the problems that the number of the data tables is increased along with the increase of the data amount, the single server cannot support the storage and calculation requirements are solved, and the storable amount of the data is expanded. And determining a target storage node in second target nodes associated with each data table to be stored according to the weight of the target node and the table information of the data table to be stored, storing the data table to be stored to the target storage node, and determining the target storage node by combining the characteristics of the data table to be stored, so that a storage mode taking the data table as a unit is realized, and the pertinence of data storage is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
Fig. 1 is a flowchart of a data storage method according to an embodiment of the present invention;
fig. 2 is a flowchart of a data storage method according to a second embodiment of the present invention;
fig. 3 is a flowchart of a data storage method according to a third embodiment of the present invention;
fig. 4 is a flowchart of a data storage method according to a fourth embodiment of the present invention;
fig. 5 is a first schematic structural diagram of a data storage device according to a fifth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a data storage device according to a fifth embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device for implementing the data storage method according to the first to third embodiments or the fourth embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," "target," and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1 is a flowchart of a data storage method according to an embodiment of the present invention, where this embodiment is applicable to a case where data tables are stored in multiple servers, and the method may be executed by a data storage device according to an embodiment of the present invention, where the data storage device may be implemented by software and/or hardware. Referring to fig. 1, the data storage method provided in this embodiment includes:
step 110, in response to the storage request, a first target node of the candidate nodes is determined.
The storage request is used to request to store data to be stored, and is sent by a storage requester, where the storage requester may be a terminal that can be connected to the candidate node, such as a computer or a mobile phone, and this embodiment is not limited thereto.
The candidate nodes are one or more online servers, each server is a node, the first target node is a node which is used as a main service node during data storage operation, when the candidate nodes comprise a plurality of nodes, a cluster is formed, a connection relation exists among the nodes, and any candidate node can be determined as the first target node.
After the first target node is determined, the data to be stored is transmitted to the first target node by the storage requester.
Step 120, obtaining target node information of each second target node in the candidate nodes through the first target node, and calculating target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function.
Among them, the candidate nodes can be divided into two types: the type I is a node with a computing function and without a storage function, and is marked as a computing type node, and if the node needs data during computing, the node can be obtained from other nodes with the storage function; the second type is a node with a computing function and a storage function, and is recorded as a computing storage type node, so that the first target node can be any one of the computing type node and the computing storage type node, and the second target node is the computing storage type node. The first target node may also be a second target node, that is, if the first target node includes a storage function, the first target node may participate in the calculation of the target node weight as the second target node.
If a plurality of second target nodes exist, target node information of all the second target nodes is acquired, that is, target node information of all nodes having a storage function is acquired, where the target node information may be acquired from a target node information manager, for example, a zookeeper or other distributed coordination service, and may include the number of CPU cores, the total amount of node memory, and the like, which is not limited in this embodiment.
For example, the manner of managing the target node information of each node through the distributed coordination service may include: each node registers target node information to the distributed coordination service, and software on each node reports the network state of the node in real time in a heartbeat mechanism and other modes with the distributed coordination service at regular time to ensure that the node is in an online state; and if the network communication between the node and the distributed coordination service is disconnected, clearing the target node information registered on the distributed coordination service, and ensuring the effectiveness of the target node information managed by the distributed coordination service.
And calculating a target node weight of each second target node according to the target node information, wherein the target node weight is the node weight calculated according to the target node information, and may be the weight of each second target node determined according to the target node information and a preset calculation formula, for example, the second target node weight is higher when the number of CPU cores is larger and the total amount of the node memory is larger.
The target node weight W of each second target node may be determined according to the following calculation formula i ′：
W i ′＝(C i ×W ci +M i ×W mi )×W i
Wherein, C i 、W ci 、M i 、W mi And W ci Are all target node information, W ci CPU weight, W, preconfigured for the second target node i mi Memory weight, W, preconfigured for second target node i i Node weights pre-configured for a second target node i; c i Is the actual CPU core number, M, of the second target node i i Is the actual total memory amount of the second target node i. By combining the target node information, the determination of the weight is based on the fact that the determination of the weight comprises various information related to the storage capacity of the second target node, and the effectiveness of the determination of the weight of the target node is improved.
And step 130, determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, and storing the data table to be stored to the target storage node.
The to-be-stored data table is a table containing to-be-stored data, the to-be-stored data needs to be stored in a table form in a specific application scenario, and the table information is information of the to-be-stored data table, such as table name information, and the like, which is not limited in this embodiment.
And determining a target storage node in second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, if the table name information is in a digital form, calculating the table name and each target node weight according to a preset formula to obtain a calculation result, and determining the second target node associated with the maximum calculation result as the target storage node.
The data to be stored can be stored to the data table to be stored through the first target node, after the target storage node is determined, the first target node informs the target storage node to download the corresponding data table to be stored from the first target node so as to store the data table to be stored, and the first target node can delete the data table stored by the target storage node, so that the storage space is prevented from being occupied.
In this embodiment, optionally, determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored includes:
determining an identification operation result according to the data table identification in the table information and the node identification of each second target node;
determining candidate node calculation results according to the identification calculation results and the target node weight, and determining target node calculation results from the candidate node calculation results according to a preset node calculation result determination mode;
and determining a target storage node associated with the calculation result of the target node according to the preset association relationship between the calculation result of the target node and the node identifier.
Each data table usually has a unique data table identifier, and can be determined according to data stored in the data table and is a numeric string with fixed digits; each node corresponds to a unique node identification, such as a predetermined node number.
And the data table identification in the table information and the node identification of each second target node can be respectively subjected to AND operation, and the identification operation result corresponding to each second target node is determined.
Determining a candidate node calculation result according to the identification calculation result and the target node weight, and taking the node calculation results of all the second target nodes as the candidate node calculation results by multiplying the identification calculation result corresponding to the second target node and the target node weight corresponding to the second target node.
The target node calculation result is determined from the candidate node calculation results according to the preset node calculation result determination manner, which may be a maximum value in the candidate node calculation results as the target node calculation result, and this embodiment does not limit this.
Illustratively, the target node calculation result N' may be determined by the following formula:
N′＝max{(R∧N i )×W i ′}
wherein R is a data table identifier, N i Is the node identification of the second target node i, W i ' is the target node weight.
The preset association relationship between the target node calculation result and the node identifier may be a mapping relationship between the target node calculation result and a node settlement result range corresponding to the node identifier, for example, if the target node calculation result is in the node calculation result range a, the target node calculation result is mapped to the node identifier one, and if the target node calculation result is in the node calculation result range B, the target node calculation result is mapped to the node identifier two. And determining a second target node with a preset association relation with the target node settlement result as a target storage node.
Because the data table identifications corresponding to different data tables to be stored are usually different, the identification operation results corresponding to different data tables to be stored are usually different, and the randomness of the calculation result determination of the target node is increased; because the candidate node calculation result is determined according to the identification calculation result and the target node weight, the second target node with the larger target node weight is easier to be determined as the target storage node, the accuracy of determining the target storage node is improved, the data table to be stored is distributed in each second target node relatively and evenly, and the balance of data storage is improved.
According to the technical scheme provided by the embodiment, the target node information of each second target node in the candidate nodes is obtained through the first target node in the candidate nodes, the target node weight of each second target node is obtained through calculation according to the target node information, the node which can store the data table to be stored in the second target node is determined according to the target node weight, the problems that the number of the data table is increased along with the increase of the data amount, the requirement of storage and calculation cannot be supported by a single server are solved, and the storable amount of the data is expanded. And determining a target storage node in a second target node associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, storing the data table to be stored to the target storage node, and determining the target storage node by combining the characteristics of the data table to be stored, so that a storage mode taking the data table as a unit is realized, and the pertinence of data storage is improved.
Example two
Fig. 2 is a flowchart of a data storage method according to a second embodiment of the present invention, and this technical solution is supplementary explained for a process after a data table to be stored is stored in a target storage node. Compared with the scheme, the scheme is specifically optimized in that after the data table to be stored is stored in the target storage node, the method further comprises the following steps:
responding to a query request of a query end, and determining a third target node in the candidate nodes; the query request comprises a first key parameter generated by a query end and data information of data to be queried;
determining a target storage node according to the data information and the target node weight, and transmitting a data table to be queried, which contains data to be queried, in the target storage node to a third target node;
acquiring data to be queried from the data table to be queried through a third target node, and performing encryption operation on the data to be queried through the first key parameter to obtain encrypted query data;
and sending the encrypted query data to the query end through the third target node so that the query end performs decryption operation on the encrypted query data through the second key parameter generated by the query end. Specifically, the flow chart of the data storage method is shown in fig. 2:
step 210, in response to the storage request, determining a first target node in the candidate nodes.
Step 220, acquiring target node information of each second target node in the candidate nodes through the first target node, and calculating target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function.
And step 230, determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, and storing the data table to be stored to the target storage node.
Step 240, responding to the query request of the query end, and determining a third target node in the candidate nodes; the query request comprises a first key parameter generated by a query end and data information of data to be queried.
The query request is used to request to query the data stored in the node, and is sent by the query end, where the query end may be a terminal that can be connected to the candidate node, such as a computer and a mobile phone, and this embodiment is not limited thereto.
The third target node is a node serving as a main service node when data query operation is performed, and any node in the candidate nodes can be determined as the third target node.
The query request includes a first key parameter generated by the query end, such as a temporary public key. The present embodiment does not limit this. The data to be queried is data to be queried corresponding to the query request, and the data information may include table information of a data table to which the data to be queried belongs, and specific position information of the data in the table.
And step 250, determining a target storage node according to the data information and the target node weight, and transmitting a data table to be queried, which contains data to be queried, in the target storage node to a third target node.
If the data information includes table information of a data table to which the data to be queried belongs, the target storage node may be determined according to the data information and the target node weight in step 230.
The target storage node transmits the data table to be queried containing the data to be queried to a third target node, and if the third target node contains the data table to be queried, the transmitting step can be omitted.
If the data to be queried exists in the transfer process, the target storage node encrypts the data table to be queried by acquiring a pre-generated public key from the third target node, and after the encrypted data table to be queried is sent to the third target node, the encrypted data table to be queried is decrypted by a private key at the third target node, so that the security of the data in the transfer process is improved.
And step 260, acquiring the data to be queried from the data table to be queried through the third target node, and encrypting the data to be queried through the first key parameter to obtain encrypted query data.
The third target node can extract the data to be queried from the data table to be queried according to the data position information in the data information, and the data to be queried is stored in a non-plaintext mode during data storage, so that the data to be queried can be extracted from the data table to be queried through preset cryptography processing, the data to be queried before and after extraction can be guaranteed to be still non-plaintext data relative to the third target node, and the security of data query is improved.
And performing encryption operation on the extracted data to be queried through the first key parameter, for example, encrypting the data to be queried through a temporary public key to obtain encrypted query data.
And 270, sending the encrypted query data to the query end through the third target node, so that the query end performs decryption operation on the encrypted query data through the second key parameter generated by the query end.
The second key parameter generated by the query end corresponds to the first key parameter, such as a temporary private key; and the third target node sends the encrypted query data to the query end so that the query end decrypts the encrypted query data through the second key parameter to obtain decrypted query data.
In this embodiment, optionally, the method further includes:
if the node adding event of the second target node exists, updating the target node information to obtain the current node information;
and calculating the current node weight of each second target node according to the current node information.
The node addition event of the second target node is an event for adding the second target node, which indicates that the nodes including the storage function are added, so that the node information of the newly added second target node can be added by the target node information manager to update the whole target node information to obtain the current node information.
And calculating to obtain the current node weight of each second target node according to the current node information, namely calculating the node weights of all the second target nodes again according to the current node information in a node weight calculation mode to obtain the current node weight.
After the second target node is added, the target storage node corresponding to the newly stored data table can be determined according to the current node weight; likewise, a new query request may target a storage node by the current node weight.
By updating the target node information in time after the second target node is newly added, the newly added node can participate in data storage in time, resource waste is avoided, and the accuracy of determining the target storage node is improved.
In this embodiment, optionally, the method further includes:
if the node deletion event of the second target node exists, updating the target node information and determining a deletion node associated with the node deletion event;
migrating the stored data associated with the pruned node to a fourth target node of the remaining nodes of the second target node;
and generating a migration relation according to the deleted node and the fourth target node, and storing the migration relation to the fourth target node in a linked list form.
The node deletion event of the second target node is an event for reducing the second target node, indicating that the nodes including the storage function are reduced, and thus the node information of the reduced second target node can be deleted by the target node information manager to update the entire target node information. And the deleted node associated with the node deletion event is the second target node which is actually reduced.
Migrating the storage data associated with the deleted nodes to a fourth target node in the rest nodes of the second target node, wherein the rest nodes are nodes which are still in a working state in the second target node after the deleted nodes, and the fourth target node is a node to which the storage data are actually migrated; the fourth target node may be determined in the following manner: after the average migration of the storage data, determining the migrated node as a fourth target node, or re-determining the weight of the target node according to the updated target node information, and calculating to obtain the fourth target node corresponding to the migrated storage data according to the re-determined weight of the target node, which is not limited in this embodiment; therefore, data loss caused by node deletion is avoided, and the safety of data storage is improved.
The migration relationship is generated according to the deleted node and the fourth target node, wherein the migration relationship comprises the deleted node where the migrated storage data originally is located and the migrated fourth target node, and the migration relationship is stored to the fourth target node in a linked list mode, so that if the position of the storage data needs to be changed due to the change of the fourth target node, a new position change relationship can be added to the linked list, the position change process of the storage data can be traced conveniently, the situation that the data cannot be obtained due to the change of the position under the conditions of data query and the like is avoided, and the orderliness of data storage is improved.
According to the embodiment of the invention, the target storage node is determined according to the data information and the target node weight, so that the situation that all nodes need to be traversed when data is inquired is avoided, and the efficiency of data inquiry is improved. The data to be inquired is encrypted through the first key parameter to obtain encrypted inquiry data, and the inquiry end decrypts the encrypted inquiry data through the second key parameter generated by the inquiry end, so that data leakage in the process of sending the data to be inquired to the inquiry end is avoided, and the safety of data transmission is improved.
EXAMPLE III
Fig. 3 is a flowchart of a data storage method according to a third embodiment of the present invention, and this technical solution is supplementary described with respect to a process of storing a data table to be stored in a target storage node. Compared with the scheme, the scheme is specifically optimized in that the data table to be stored is stored to the target storage node, and comprises the following steps:
acquiring data table identification and data table quantity of a data table to be stored corresponding to a target storage node;
and creating a target directory associated with each data table to be stored at the target storage node according to the number of the data tables and a preset directory table number threshold, and storing the data table to be stored to the target directory. Specifically, the flow chart of the data storage method is shown in fig. 3:
step 310, in response to the storage request, a first target node of the candidate nodes is determined.
Step 320, obtaining target node information of each second target node in the candidate nodes through the first target node, and calculating target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function.
And step 330, determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored.
And 340, acquiring the data table identification and the data table quantity of the data table to be stored corresponding to the target storage node.
The data tables to be stored corresponding to the target storage nodes are data tables which are determined to be stored in the target storage nodes, the corresponding target storage nodes can be determined for all the data tables to be stored, and the data table identification and the data table number of the data tables to be stored in each target storage node are obtained; illustratively, 10000 data tables to be stored coexist, where the data tables 1-1000 to be stored are allocated to the first target storage node, the number of the data tables of the first target storage node is 1000, and the data table is identified as the identity of this thousand data tables.
And 350, creating a target directory associated with each data table to be stored at the target storage node according to the number of the data tables and a preset directory table number threshold, and storing the data table to be stored to the target directory.
Target directories are created at the target storage nodes, and each target directory can contain a corresponding data table.
The preset directory table number threshold is a maximum number value of tables that can be accommodated in a preset single directory, for example 10000, which is not limited in this embodiment.
Determining the number of directories n1, n1= S1/M1 according to the number of the data tables and a preset threshold of the number of the directories, wherein S1 is the number of the data tables; m1 is a preset directory table quantity threshold value and can be adjusted according to the disk read-write performance of the node.
The directory identifier of each directory can be determined according to the directory number and the data table identifier, the directory identifier includes a part of each data table identifier, and the target directory is determined according to the data table identifier and the directory identifier, and the data table identifiers are 101878, 101567, 110678, 110598 and 110908 by way of example. If the number of the directories is 2, it indicates that 5 data tables need to be stored in two directories, and for storage balance, the tables 101878 and 101567 can be stored in directory one, and the directory identifier is determined to be 101; storing the tables 110678, 110598 and 110908 into a directory two, and determining the directory identification as 110; therefore, the target directory corresponding to table 101878 is directory 101.
It should be noted that, for storage balance, the number of the directories and the number of bits of the directory identifier corresponding to the data table identifier may be different, for example, if the number of the directories is 1, the directory identifier may be 1, which is not limited in this embodiment.
After the target directory is determined, the target directory can be created in the target storage node, so that the data tables to be stored are respectively stored in the corresponding target directories.
The database type of the storage database of the target storage node may be an LMDB, so as to improve access efficiency to the storage data, and accordingly, the target directory may be constructed based on the LMDB, which is not limited in this embodiment.
According to the embodiment of the invention, the target directories associated with the data tables to be stored are established in the target storage nodes according to the number of the data tables and the preset directory table number threshold, and the data tables to be stored are stored in the target directories, so that the storage orderliness is improved, the data storage capacity of a single node is improved, and the problem that the data tables to be stored are directly stored in the target storage nodes, so that the data tables to be stored are excessive in number during subsequent access, and if the data tables are sequentially inquired, the data access efficiency is reduced, and even the database is possibly dead; when the data table is accessed subsequently, initial access can be performed through the directory identification, and then further access is performed from the accessed directory, so that the data access efficiency is improved.
Example four
Fig. 4 is a flowchart of a data storage method according to a fourth embodiment of the present invention, where this embodiment is applicable to a case where a data table is stored in a single server, and the method may be executed by a data storage device according to the fourth embodiment of the present invention, where the data storage device may be implemented by software and/or hardware. Referring to fig. 4, the data storage method provided in this embodiment includes:
and step 410, storing the data to be stored into the data table to be stored, and generating the data table identification and the data table quantity.
After the data to be stored is stored in the data table in a preset storage mode, different data tables can generate corresponding data table identifications according to specific data stored in the data tables in a preset identification calculation mode; the preset storage manner may be to store data of the same data type into the same table, and the like, which is not limited in this embodiment.
And after all the data to be stored are stored in the table, the data size of the data table can be obtained.
And step 420, creating a target directory associated with each data table to be stored in the current server according to the number of the data tables and a preset directory table number threshold, and storing the data table to be stored in the target directory.
The current server is a server for storing data currently, a target directory is created in the current server, and each target directory can contain a corresponding data table.
The preset directory table number threshold is a maximum number value of tables that can be accommodated in a preset single directory, for example 10000, which is not limited in this embodiment.
Determining the number of directories n2, n2= S2/M2 according to the number of the data tables and a preset number threshold of the directory tables, wherein S2 is the number of the data tables; m2 is a preset directory table quantity threshold value and can be adjusted according to the disk read-write performance of the current server.
The directory identifier of each directory can be determined according to the directory number and the data table identifier, the directory identifier includes a part of each data table identifier, and the target directory is determined according to the data table identifier and the directory identifier, and the data table identifiers are 101878, 101567, 110678, 110598 and 110908 by way of example. If the number of the directories is 2, it indicates that 5 data tables need to be stored in two directories, and for storage balance, the tables 101878 and 101567 can be stored in directory one, and the directory identifier is determined to be 101; storing the tables 110678, 110598 and 110908 into a directory two, and determining the directory identification as 110; therefore, the target directory corresponding to table 101878 is directory 101.
It should be noted that, for storage balance, the number of the directories and the number of bits of the directory identifier corresponding to the different identifiers of the data table may be different, for example, if the number of the directories is 1, the directory identifier may be 1, which is not limited in this embodiment.
After the target directory is determined, the target directory can be created in the current server, so that the data tables to be stored are respectively stored in the corresponding target directories.
The database type of the storage database of the current server may be an LMDB, so as to improve access efficiency to the storage data, and accordingly, the target directory may be constructed based on the LMDB, which is not limited in this embodiment.
Optionally, if the number of the data tables is greater than the preset threshold, a sub-directory may be established in the directory for further storage.
In the prior art, a general relational database and a non-relational database system, such as MySQL, mongoDB, and the like, are usually adopted in a server to directly store data; if data access is performed when the data volume or the data tables are excessive, the data access efficiency is reduced, and even the database may be dead. According to the technical scheme provided by the embodiment, data are written in a table dividing mode, the data are stored in a table form, the requirement of a service scene is met, when the number of the data tables is large, a target directory associated with each data table to be stored is created in the current server according to the number of the data tables and a preset directory table number threshold, the data table to be stored is stored in the target directory, the storage orderliness is improved, the problems in the prior art are solved, and the data storage capacity of a single server is improved. When the data table is accessed subsequently, the initial access can be carried out through the directory identification, and then the further access can be carried out from the accessed directory, so that the data access efficiency is improved.
EXAMPLE five
Fig. 5 is a first schematic structural diagram of a data storage device according to a fifth embodiment of the present invention. The device can be realized by hardware and/or software, can execute the data storage method provided by the first to third embodiments of the invention, and has the corresponding functional modules and beneficial effects of the execution method. As shown in fig. 5, the apparatus includes:
a first node determining module 510 for determining a first target node among the candidate nodes in response to the storage request;
a first weight calculation module 520, configured to obtain, by the first target node, target node information of each second target node in the candidate nodes, and calculate, according to the target node information, a target node weight of each second target node; wherein the second target node comprises a storage function;
a first data table storage module 530, configured to determine, according to the target node weight and table information of the data tables to be stored, a target storage node in the second target node associated with each data table to be stored, and store the data table to be stored to the target storage node.
On the basis of the foregoing technical solutions, optionally, the first data table storage module includes:
an operation result determining unit, configured to determine an identification operation result according to a data table identifier in the table information and a node identifier of each second target node;
the target node calculation result determining unit is used for determining candidate node calculation results according to the identification calculation results and the target node weights, and determining target node calculation results from the candidate node calculation results according to a preset node calculation result determining mode;
and the target storage node determining unit is used for determining the target storage node associated with the target node calculation result according to the preset association relationship between the target node calculation result and the node identifier.
On the basis of the above technical solutions, optionally, the apparatus further includes:
the third node determining module is used for responding to a query request of a query end after the data table storage module and determining a third target node in the candidate nodes; the query request comprises a first key parameter generated by the query end and data information of data to be queried;
the data transmission module is used for determining the target storage node according to the data information and the target node weight and transmitting a data table to be queried, which contains the data to be queried, in the target storage node to the third target node;
the data encryption module is used for acquiring the data to be queried from the data table to be queried through the third target node and encrypting the data to be queried through the first key parameter to obtain encrypted query data;
and the data decryption module is used for sending the encrypted query data to the query end through the third target node so that the query end decrypts the encrypted query data through the second key parameter generated by the query end.
On the basis of the above technical solutions, optionally, the apparatus further includes:
a node information updating module, configured to update the target node information to obtain current node information if a node increase event of the second target node exists;
and the second weight calculation module is used for calculating the current node weight of each second target node according to the current node information.
On the basis of the above technical solutions, optionally, the apparatus further includes:
a pruned node determining module, configured to update the target node information if a node pruned event of the second target node exists, and determine a pruned node associated with the node pruned event;
a data migration module for migrating the stored data associated with the pruned node to a fourth target node of the remaining nodes of the second target node;
and the relationship storage module is used for generating a migration relationship according to the deleted node and the fourth target node, and storing the migration relationship to the fourth target node in a linked list form.
On the basis of the foregoing technical solutions, optionally, the first data table storage module includes:
the identification number determining unit is used for acquiring the data table identification and the data table number of the data table to be stored corresponding to the target storage node;
and the data table storage unit is used for creating a target directory associated with each data table to be stored in the target storage node according to the number of the data tables and a preset directory table number threshold value, and storing the data table to be stored in the target directory.
Fig. 6 is a schematic structural diagram of a data storage device according to a fifth embodiment of the present invention. The device can be realized in a hardware and/or software mode, can execute the data storage method provided by the fourth embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. As shown in fig. 6, the apparatus includes:
the data storage module 610 is configured to store data to be stored in a data table to be stored, and generate a data table identifier and a data table number;
and a second data table storage module 620, configured to create, in the current server, a target directory associated with each data table to be stored according to the number of the data tables and a preset directory table number threshold, and store the data table to be stored in the target directory.
EXAMPLE six
Fig. 7 is a schematic structural diagram of an electronic device that can be used to implement the data storage method in embodiments one to three or embodiment four of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 7, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 executes the respective methods and processes described above, such as the data storage method in the embodiment one to three or the data storage method in the embodiment four.
In some embodiments, the data storage method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data storage method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data storage method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A method of storing data, comprising:
determining a first target node of the candidate nodes in response to the storage request;
acquiring target node information of each second target node in the candidate nodes through the first target node, and calculating target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function;
and determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, and storing the data table to be stored to the target storage node.
2. The method of claim 1, wherein determining a target storage node of the second target nodes associated with each of the data tables to be stored according to the target node weight and table information of the data tables to be stored comprises:
determining an identification operation result according to a data table identification in the table information and the node identification of each second target node;
determining candidate node calculation results according to the identification calculation results and the target node weight, and determining target node calculation results from the candidate node calculation results according to a preset node calculation result determination mode;
and determining the target storage node associated with the target node calculation result according to the preset association relationship between the target node calculation result and the node identifier.
3. The method according to claim 1, further comprising, after storing the data table to be stored to the target storage node:
responding to a query request of a query end, and determining a third target node in the candidate nodes; the query request comprises a first key parameter generated by the query end and data information of data to be queried;
determining the target storage node according to the data information and the target node weight, and transmitting a data table to be queried, which contains the data to be queried, in the target storage node to the third target node;
acquiring the data to be queried from the data table to be queried through the third target node, and encrypting the data to be queried through the first key parameter to obtain encrypted query data;
and sending the encrypted query data to the query end through the third target node, so that the query end decrypts the encrypted query data through a second key parameter generated by the query end.
4. The method of claim 1, further comprising:
if the node adding event of the second target node exists, updating the target node information to obtain current node information;
and calculating the current node weight of each second target node according to the current node information.
5. The method of claim 1, further comprising:
if the node deletion event of the second target node exists, updating the target node information and determining a deletion node associated with the node deletion event;
migrating the stored data associated with the pruned node to a fourth target node of the remaining nodes of the second target node;
and generating a migration relationship according to the deleted node and the fourth target node, and storing the migration relationship to the fourth target node in a linked list form.
6. The method according to any one of claims 1 to 5, wherein storing the data table to be stored to the target storage node comprises:
acquiring data table identification and data table quantity of a data table to be stored corresponding to the target storage node;
and creating a target directory associated with each data table to be stored at the target storage node according to the number of the data tables and a preset directory table number threshold, and storing the data table to be stored to the target directory.
7. A method of storing data, comprising:
storing the data to be stored into a data table to be stored, and generating a data table identifier and the number of the data tables;
and creating a target directory associated with each data table to be stored in the current server according to the number of the data tables and a preset directory table number threshold, and storing the data tables to be stored in the target directory.
8. A data storage device, comprising:
a first node determination module to determine a first target node of the candidate nodes in response to the storage request;
the first weight calculation module is used for acquiring target node information of each second target node in the candidate nodes through the first target node and calculating the target node weight of each second target node according to the target node information; wherein the second target node comprises a storage function;
and the first data table storage module is used for determining a target storage node in the second target nodes associated with each data table to be stored according to the target node weight and the table information of the data table to be stored, and storing the data table to be stored to the target storage node.
9. A data storage device, comprising:
the data storage module is used for storing the data to be stored into a data table to be stored and generating data table identification and data table quantity;
and the second data table storage module is used for creating a target directory associated with each data table to be stored in the current server according to the number of the data tables and a preset directory table number threshold value, and storing the data table to be stored in the target directory.
10. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data storage method of any one of claims 1-6 or claim 7.
11. A computer-readable storage medium storing computer instructions for causing a processor to perform the data storage method of any one of claims 1 to 6 or claim 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211708814.1A CN115858539A (en) | 2022-12-29 | 2022-12-29 | Data storage method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211708814.1A CN115858539A (en) | 2022-12-29 | 2022-12-29 | Data storage method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115858539A true CN115858539A (en) | 2023-03-28 |
Family
ID=85655950
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211708814.1A Pending CN115858539A (en) | 2022-12-29 | 2022-12-29 | Data storage method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115858539A (en) |
-
2022
- 2022-12-29 CN CN202211708814.1A patent/CN115858539A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9996565B2 (en) | Managing an index of a table of a database | |
| US10592532B2 (en) | Database sharding | |
| CN109800270B (en) | Data storage and query method and Internet of things system | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| JP2021525907A (en) | Frequent pattern analysis of distributed systems | |
| CN114253979A (en) | Message processing method and device and electronic equipment | |
| CN112528067A (en) | Graph database storage method, graph database reading method, graph database storage device, graph database reading device and graph database reading equipment | |
| CN112148728A (en) | Method, apparatus and computer program product for information processing | |
| CN115858539A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN115099175A (en) | Method and device for acquiring time sequence netlist, electronic equipment and storage medium | |
| JP2016045594A (en) | Data processing apparatus, data processing method, and data processing program | |
| US11151110B2 (en) | Identification of records for post-cloning tenant identifier translation | |
| CN113778645A (en) | Task scheduling method, device and equipment based on edge calculation and storage medium | |
| CN112948458B (en) | Block chain-based query method and device | |
| CN117056133B (en) | Data backup method, device and medium based on distributed Internet of things architecture | |
| CN116760762B (en) | Decentralised ad hoc network method and device | |
| CN114650222B (en) | Parameter configuration method, device, electronic equipment and storage medium | |
| CN113220282B (en) | Development processing method, device and storage medium for applet | |
| CN115525659A (en) | Data query method and device, electronic equipment and storage medium | |
| CN113946702A (en) | Image feature storage method and device, storage medium and electronic equipment | |
| CN113434508A (en) | Method and apparatus for storing information | |
| CN117112576A (en) | Data synchronization method and device, server and storage medium | |
| CN115408357A (en) | File record processing method, device, equipment and medium | |
| CN117632968A (en) | Data writing method, device, equipment and medium based on distributed database | |
| CN116701220A (en) | Data synchronization test method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |