CN110737663A - data storage method, device, equipment and storage medium - Google Patents
data storage method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110737663A CN110737663A CN201910979709.3A CN201910979709A CN110737663A CN 110737663 A CN110737663 A CN 110737663A CN 201910979709 A CN201910979709 A CN 201910979709A CN 110737663 A CN110737663 A CN 110737663A
- Authority
- CN
- China
- Prior art keywords
- node
- target object
- data
- target
- fragment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 60
- 238000000034 method Methods 0.000 title claims abstract description 57
- 239000012634 fragment Substances 0.000 claims abstract description 100
- 238000013467 fragmentation Methods 0.000 claims description 47
- 238000006062 fragmentation reaction Methods 0.000 claims description 47
- 238000010276 construction Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 238000002955 isolation Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000001152 differential interference contrast microscopy Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The application discloses data storage methods, devices, equipment and storage media, wherein the method comprises the steps of receiving a storage request of data in a target object, wherein the storage request carries attribute information of the target object, determining a routing key of the target object according to the attribute information of the target object, determining a fragment number of the target object according to the routing key of the target object, querying a fragment tree according to the fragment number of the target object, determining a target storage node of the data in the target object, and storing the data in the target object based on the target storage node.
Description
Technical Field
The present application relates to the field of internet communications technologies, and in particular, to data storage methods, apparatuses, devices, and storage media.
Background
In order to adapt to the increase of the data volume, the problem of the data volume can be solved only by continuously optimizing the database.
Therefore, it is necessary to provide data storage methods, apparatuses, devices, and storage media, to implement fragmented storage of data, and perform individual fragmented storage of data of important users, so as to accelerate data reading speed and facilitate users to quickly query data.
Disclosure of Invention
The application provides data storage methods, devices, equipment and storage media, which can realize fragmented storage of data and separate fragmented storage of data of important users, thereby accelerating the reading speed of data and facilitating the quick query of data by users.
, the present application provides a method of data storage, the method comprising:
receiving a storage request of data in a target object, wherein the storage request carries attribute information of the target object;
determining a routing key of the target object according to the attribute information of the target object;
determining the fragment number of the target object according to the routing key of the target object;
inquiring a fragment tree according to the fragment number of the target object, and determining a target storage node of data in the target object;
storing data in the target object based on the target storage node;
the construction method of the fragmentation tree comprises the following steps:
, the fragment number of the sample object and the fragment number of the second sample object are obtained;
taking the last N-bit character of the slicing number of the -th sample object as identification information of a -th node, wherein N is a positive integer;
taking the fragment number of the second sample object as identification information of a second node;
and constructing the sharding tree based on the identification information of the th node and the identification information of the second node.
Another aspect provides a data storage device comprising:
a storage request receiving module, configured to receive a storage request for data in a target object, where the storage request carries attribute information of the target object;
a routing key determining module, configured to determine a routing key of the target object according to the attribute information of the target object;
the fragment number determining module is used for determining the fragment number of the target object according to the routing key of the target object;
the target storage node determining module is used for inquiring the fragment tree according to the fragment number of the target object and determining the target storage node of the data in the target object;
the data storage module is used for storing the data in the target object based on the target storage node;
the fragment number acquisition module is used for acquiring the fragment number of the th sample object and the fragment number of the second sample object;
an th identification information determining module, configured to use the last N-bit character of the slice number of the th sample object as identification information of a th node, where N is a positive integer;
a second identification information determining module, configured to use the fragment number of the second sample object as identification information of a second node;
and the fragmentation tree construction module is used for constructing a fragmentation tree based on the identification information of the th node and the identification information of the second node.
A further aspect provides a data storage device comprising a processor and a memory having stored therein at least instructions, at least programs, sets of codes, or sets of instructions, said at least instructions, said at least programs, sets of codes, or sets of instructions being loaded and executed by said processor to implement the data storage method as described above.
A further aspect provides computer readable storage media having stored therein at least instructions, at least program segments, code sets, or instruction sets, the at least instructions, the at least program segments, code sets, or instruction sets being loaded and executed by a processor to implement the data storage methods described above.
The data storage method, the data storage device, the data storage equipment and the data storage medium have the following technical effects:
according to the method and the device, the fragmentation tree capable of realizing suffix matching and accurate matching of the node identification information is constructed, the routing key of the target object is determined through the attribute information of the target object, the fragmentation number of the target object is determined in step , the fragmentation tree is inquired through the fragmentation number, the target storage node is determined, fragmentation storage of data is realized, and independent fragmentation storage of data of important users can be realized, so that the data reading speed is increased, and the users can conveniently and quickly inquire the data.
Drawings
In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of an data storage system provided by an embodiment of the present application;
FIG. 2 is a schematic flow chart of an data storage method according to an embodiment of the present application;
FIG. 3 is a schematic flow chart diagram illustrating a method for determining a target storage node of data in a target object according to an embodiment of the present application;
FIG. 4 is a schematic flow chart diagram illustrating a method for constructing child nodes of a target storage node according to exemplary embodiments of the present application;
FIG. 5 is a flow chart of methods for building a shard tree according to embodiments of the present disclosure;
fig. 6 is a schematic structural diagram of sharded tree nodes provided in an embodiment of the present application;
FIG. 7 is a schematic structural diagram of kinds of partition trees provided in the embodiments of the present application;
FIG. 8 is a schematic structural diagram of another kinds of partition trees provided in the embodiments of the present application;
FIG. 9 is a schematic structural diagram of storage nodes provided in an embodiment of the present application;
FIG. 10 is a schematic structural diagram of an data storage device provided by an embodiment of the present application;
fig. 11 is a schematic structural diagram of servers provided in the embodiment of the present application;
FIG. 12 is a schematic structural diagram of an blockchain system according to an embodiment of the present invention;
fig. 13 is a block structure diagram according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application , rather than all embodiments.
Furthermore, the terms "comprises" and "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a series of steps or elements of is not necessarily limited to the expressly listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, fig. 1 is a schematic diagram of an data storage system according to an embodiment of the present application, and as shown in fig. 1, the data storage system may include at least a server 01 and a client 02.
In particular, in the embodiment of the present specification, the server 01 may include servers operating independently, or a distributed server, or a server cluster composed of a plurality of servers, the server 01 may include a network communication unit, a processor, a memory, and the like, and in particular, the server 01 may be used for storing data.
Specifically, in the embodiment of the present disclosure, the client 02 may include a physical device of a type such as a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, a smart wearable device, and the like, and may also include software running in the physical device, such as a web page provided by service providers to a user, and an application provided by the service providers to the user.
The terms referred to in the embodiments of the present specification are explained as follows.
Entity: entity, a data storage object, such as a relational database or a table of non-relational data;
data Sharding, storing Data distributed according to certain characteristic information to a plurality of physical storage nodes or virtual nodes, the Data Sharding of the present application is similar to horizontal partitioning (horizontal partitioning);
routing key: route Key, also called data slicing Key (shading Key), is the characteristic information stored in data slicing.
Splitting the tree into pieces: sharding Tree, data Sharding Tree. Inquiring the fragment tree according to the routing key to determine the storage position information of the data;
and (4) slicing marking: flag or shading Flag, data fragmentation Flag information, is identification information of a data storage node or a database instance.
The following describes data storage methods of the present application, and fig. 2 is a flow chart of data storage methods provided by the embodiments of the present application, and the present specification provides the method operation steps as described in the embodiments or the flow chart, but may include more or less operation steps based on conventional or non-inventive labor, the step sequence listed in the embodiments is only ways of the execution sequence of many steps, and does not represent the execution sequence of only .
S201: receiving a storage request of data in a target object, wherein the storage request carries attribute information of the target object.
In this embodiment of the present specification, an execution subject of the method may be a server, and the storage request of the data in the target object may be sent to the server by a user through a terminal.
In this embodiment of the present specification, the target object may be a storage entity of data, for example, the target object may be a database instance or a database table of a relational database, and the attribute information of the target object may include identification information of a user and a generation time of the data. The attribute information of the target object may be an attribute field (KeyField) of the target object, and the attribute field is determined based on the data storage entity object.
S203: determining a routing key of the target object according to the attribute information of the target object;
in this embodiment of the present specification, the routing Key (Route Key), also called data fragmentation Key (shading Key), is feature information stored in a data fragmentation, and is a Key attribute Field (Key Field) for querying and routing the data fragmentation. The routing key is an attribute of a data storage Entity (Entity) and determines on which data node data is stored.
S205: determining the fragment number of the target object according to the routing key of the target object;
in the embodiment of the description, the routing key generates a Sharding Number (shading Number) after being processed by an algorithm or translation determined by , and the Sharding Number queries a matching Sharding Tree (shading Tree) to accurately locate the storage node information (flag) of data only . since the Sharding Tree is a Number Tree (shading Number Tree), the routing key which is not a Number needs to be converted into a Number string through a mechanism or algorithm determined by .
In specific embodiments, the code for converting the route key of the target object into the fragment number of the target object is as follows:
s207: inquiring a fragment tree according to the fragment number of the target object, and determining a target storage node of data in the target object;
in this embodiment, the shard tree is also referred to as a shard number tree, and the shard tree supports both end matching and precise matching of data shard key values, that is, data of a certain set or some set active clients or vip (very Important person) large clients is sharded and isolated individually.
In this embodiment of this specification, as shown in fig. 3, the querying the fragmentation tree according to the fragmentation number of the target object to determine the target storage node of the data in the target object may include:
s2071: determining a post-preset bit character in the fragment number of the target object, and taking the post-preset bit character as a current character string;
in this embodiment of the present specification, the last preset bit character in the fragment number of the target object refers to a last N bit character in the fragment number of the target object, where N may be an integer such as 1, 2, 3, … …, for example, the fragment number of the target object may be "81011", where "11" and "1011" are both corresponding last preset bit characters, and in actual application, the number of bits of a character may be determined according to actual situations.
S2073: judging whether a node matched with the current character string exists in the fragment tree or not;
in this embodiment of the present specification, the node that matches the current character string refers to a node whose identification information is identical to a character in the current character string. The identification information of the node may be a data fragmentation flag.
S2075: if the segment tree has a node matched with the current character string, increasing the length of the post-preset character, and taking the character with the increased length as the current character string; repeating the steps: judging whether a node matched with the current character string exists in the fragment tree or not;
in the embodiment of the present specification, a node that realizes longest character matching with a fragment number of a target object can be obtained by continuously increasing the length of a post-preset bit character.
In specific embodiments, the fragment number of the target object may be "81011", a node matching the character string "11" is found first, if the node exists, a node matching "1011" is found, and finally "81011" is found, so that an exact match can be achieved.
Wherein the '$' character identification in the routing configuration number is accurately matched.
In specific embodiments, the fragmentation number of the target object is "71011", a node matching the character string "11" is found first, and if the node exists, a node matching the character string "1011" is found, and since the node 71011 does not exist, the fragmentation number "71011" is matched to the node "1011", thereby realizing the longest character matching.
In an embodiment of the present specification, if a node matching the current character string exists in the segment tree, increasing the length of the post-default-bit character may include:
if the fragment tree has a node matched with the current character string, judging whether the fragment numbers of the current character string and the target object are the same;
if not, increasing the length of the post-preset bit character.
In this embodiment of this specification, after the step of determining whether the current character string is the same as the fragment number of the target object, the method may further include:
and if the current character string is the same as the fragment number of the target object, determining the node corresponding to the current character string as a target storage node of the data in the target object.
S2077, if the fragment tree does not have a node matched with the current character string, determining the character of the current character string before the last times of length increase as a target character;
in this embodiment of the present specification, when there is no node matching the current character string in the segment tree, which indicates that the target object corresponding to the current character cannot be matched exactly, the first characters corresponding to the current character are determined as target characters, and a storage node is determined.
S2079: and determining the node matched with the target character as a target storage node of the data in the target object.
In this embodiment of the present specification, a storage node of data in a target object is determined by matching or exact matching of a longest suffix of a segment number of the target object, so that data of different users is stored in different nodes.
S209: and storing the data in the target object based on the target storage node.
In the embodiments of the present specification, the data storage nodes, i.e., the data storage node information corresponding to the data fragmentation flag (flag). The data storage nodes may be actual data storage nodes such as database instances, may be virtual data storage nodes, and the like. The fragmentation flag (flag) corresponds to the address information of a specific data storage node or database instance: IP address, port number; authentication information: user name, password, etc.; and copy information, etc. The schematic structural diagram of the storage node is shown in fig. 9, where 1, 2, 3, … …, n are identification information of the storage node.
In specific embodiments, the data storage node described using the protobuf protocol is as follows:
data storage objects described using the protobuf protocol:
the shards of data may be initially fixed shards, such as 100 library 10 tables (100 shards) on a relational database, and as the system is run online, shards are added further after the suffix of the routing key in step , depending on the amount of data in each shard, and for data-active routing keys, it is also possible to individually assign a particular active routing key or keys to or more particular shards.
In an embodiment of this specification, before the step of storing the data in the target object based on the target storage node, the method may further include:
constructing child nodes of the target storage node;
correspondingly, the storing the data in the target object based on the target storage node includes:
and storing the data in the target object based on the child nodes of the target storage node.
In embodiments, as shown in fig. 4, before the step of constructing the child node of the target storage node, the method may further include:
s2081: judging whether the identification information of the target object is preset identification information or not;
in this embodiment, the preset identification information may be identification information of a VIP user or a designated user.
Accordingly, the constructing the child node of the target storage node may include:
s2083: and when the identification information of the target object is preset identification information, constructing child nodes of the target storage node.
In the embodiment of the present specification, by constructing child nodes of a target storage node, data of a special user can be stored separately, so that the data reading speed is increased, and a user can query data quickly.
In embodiments, as shown in fig. 4, before the step of constructing the child node of the target storage node, the method may further include:
s2085: judging whether the residual space of the target storage node is smaller than a preset threshold value or not;
accordingly, the constructing the child node of the target storage node may include:
s2087: and when the residual space of the target storage node is smaller than a preset threshold value, constructing child nodes of the target storage node.
In this embodiment of the present specification, when the storage space of the target storage node is insufficient, child nodes of the target storage node are constructed, that is, data can be stored by adding child nodes, and data migration can also be performed.
In an embodiment of this specification, the method for constructing the sharded tree may include:
s301, acquiring the fragment number of the th sample object and the fragment number of the second sample object;
s303, taking the last N-bit character of the slicing number of the th sample object as the identification information of the th node, wherein N is a positive integer;
s305: taking the fragment number of the second sample object as identification information of a second node;
s307, constructing the fragmentation tree based on the identification information of the th node and the identification information of the second node.
In specific embodiments, as shown in fig. 5, the method for constructing the sharding tree may include determining a sharding number of a sample object based on a route key of the sample object, where the sample object includes a th sample object and a second sample object, then determining identification information of a th node and a second node based on a th sample object and the second sample object, where the th node is configured to store data in an object corresponding to the sharding number and having N-bit characters matching the th node, where N is a positive integer, the second node is configured to store data in an object corresponding to the sharding number and having each character matching the second node, and finally constructing the sharding tree based on the identification information of the th node and the identification information of the second node, where 1-N are identification information of nodes.
In the embodiment of the present specification, the fragmentation tree supports two ways, namely, last order matching and exact matching of the routing number.
And dynamically generating and loading the fragment number tree according to the configuration tool. And (4) changing the fragment number tree, and triggering data migration (data copying and data deleting).
A Sharding Tree (Sharding Tree), which may be 10-pronged trees consisting of the numbers 0.. 9.
The sharded tree is composed of root nodes (roots) and child nodes. And configuring default storage node information on the root node.
As shown in fig. 6, each node mainly includes five pieces of information:
child nodes (next number nodes): sub-node number (number) of current node
Suffix number (suffix number): number string of path from root node to current node
data slicing flag (flag) matching current node suffix with corresponding slicing information
Second data slice flag (final _ flag): accurate matching of current node and corresponding fragment information
Valid children (next valid nodes): list of active nodes (nodes with fragmentation flag) under the current node.
In concrete embodiments, the structure of the fragment tree is shown in fig. 7, where the number inside the circle of each node is the number of the current node, the connecting line below the circle points to the immediate child node (next number nodes) of the current node, the number inside the -th box is a suffix matching data fragment flag (flag), the number inside the second box is a precisely matching data fragment flag (final _ flag), and the virtual connecting line below the third box points to the valid child node (next valid nodes) of the current node.
Routing configuration information is an attribute of a data storage entity, such as a database instance or a database table of a relational database. Relevant routing information is configured on the data storage entity. After the routing information is changed, the hot loading can take effect.
Routing configuration information described using the protobuf protocol:
in specific embodiments, the structure of the shard tree is shown in fig. 8, where the shard flag corresponding to suffix number 11 is 1, and the valid child node of number 11 is 1011;
the fragmentation mark corresponding to the node 1011 is 3, and the valid child nodes of the number 1011 are 21011, 31011, 2310841011 and 81011;
the fragmentation flag corresponding to the node 21011 is 2; the valid child node for number 21011 is 8421011;
the accurate matching shard mark corresponding to the node 2310841011 is 5; number 2310841011 has no valid child node
The accurate matching fragmentation mark corresponding to the node 81011 is 5; the valid child node of number 81011 is 2781011;
the fragmentation mark corresponding to the node 8421011 is 3; number 8421011 has no valid child node;
the accurate matching shard mark corresponding to the node 2531011 is 5; number 2531011 has no valid child node;
the accurate matching shard mark corresponding to the node 2781011 is 5; number 2781011 has no valid child node;
the construction code of the slicing tree is as follows:
wherein the '$' character identification in the routing configuration number is accurately matched.
Because of the complex fragmentation of data, both the suffix matching fragmentation of the routing key and the precise matching fragmentation of the routing key, when querying (WALK, etc.) or aggregating across fragmentation, it is necessary to traverse each node of the fragmentation tree, obtain the valid node of the next different fragmentation marks of the current node at each fragmentation node according to the suffix number tree, and then filter the data in the next different fragmentation marks under the current node.
According to the suffix number tree above, the SQL statements that traverse 11 fragmented subtrees are as follows:
slicing flag is executed on 1
SELECT*FROM entity where routekey like‘％11’AND routekey not like‘％1011’；
The meaning is as follows: the last bit of the read routing key from slice 1(flag ═ 1) contains "11", and the last bit does not contain data of "1011".
Execution statement after 3-slicing flag optimization
SELECT*FROM entity where routekey like‘％1011’
AND routekey not like‘％21011’
AND routekey<>‘31011’
AND routekey<>‘2310841011’
AND routekey<>‘81011’
AND routekey<>‘2781011’
AND routekey<>‘2531011’；
The meaning is as follows: the last bit of the routing key read from slice 3(flag ═ 3) contains "1011", and the last bit of the routing key does not contain "21011", and the routing key does not equal the data of "31011", "2310841011", "81011", "2781011", "2531011".
Slicing flag is the SQL statement executed on 2:
SELECT*FROM entity where routekey like‘％21011’
AND routekey not like‘％8421011’；
the meaning is as follows: the last bit read from slice 2(flag ═ 2) for the route key contains "21011" and the last bit of the route key does not contain the data of "8421011".
And 5, the optimized execution statement:
SELECT*FROM entity where routekey IN
(‘321011’,‘81011’,‘2310841011’，‘2781011’，‘2531011’)；
the meaning is as follows: data with routing keys equal to "321011", "81011", "2310841011", "2781011", and "2531011" are read from slice 5(flag ═ 5).
Slicing flag is the SQL statement executed on 3:
SELECT*FROM entity where routekey like‘％8421011’；
the meaning is as follows: and reading data with the final bit of '8421011' from the fragment 3(flag is 3).
The data storage method is applicable to data fragmentation and routing Query on a traditional relational database, and is also applicable to a non-relational database and a database system of NewSQL, wherein NewSQL is short for various new extensible/high-performance databases, and the databases not only have the storage management capacity of NoSQL (non-relational database) on mass data, but also maintain the characteristics of the traditional databases such as ACID (atomic, Consistency responsiveness, Isolation, dual Durability) and SQL (Structured Query Language).
In embodiments of the present description, data in a target object may be stored based on a blockchain system that includes a plurality of nodes forming a peer-to-peer network therebetween.
In , the blockchain system may be the structure shown in fig. 12, a plurality of nodes form a Peer-To-Peer (P2P, Peer To Peer) network, the P2P Protocol is application layer protocols running on top of a Transmission Control Protocol (TCP), in the blockchain system, any machine, such as a server and a terminal, may be added To become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer and an application layer.
The functions of each node in the blockchain system shown in fig. 12 include:
1) routing, a basic function that a node has, is used to support communication between nodes.
Besides the routing function, the node may also have the following functions:
2) the application is used for being deployed in a block chain, realizing specific services according to actual service requirements, recording data related to the realization functions to form recording data, carrying a digital signature in the recording data to represent a source of task data, and sending the recording data to other nodes in the block chain system, so that the other nodes add the recording data to a temporary block when the source and integrity of the recording data are verified successfully.
3) The Block chain comprises series of blocks (blocks) which are connected with each other according to the generated chronological order, a new Block cannot be removed again once being added into the Block chain, and recorded data submitted by nodes in the Block chain system are recorded in the blocks.
In embodiments, the Block Structure (Block Structure) may be the Structure shown in fig. 13, where each Block includes a hash value of the Block storing the transaction record (hash value of the Block) and hash values of the first blocks, and the blocks are connected by the hash values to form a Block chain.
As can be seen from the technical solutions provided by the embodiments of the present specification, a fragment tree capable of implementing suffix matching and precise matching of node identification information is constructed in the embodiments of the present specification, a routing key of a target object is determined according to attribute information of the target object, and a fragment number of the target object is determined, and a target storage node is determined by querying the fragment tree according to the fragment number, so that fragment storage of data is implemented, and data of an important user can be separately stored in fragments, thereby speeding up data reading and facilitating quick data query by a user.
Embodiments of the present application also provide data storage devices, as shown in fig. 10, the device may include:
a storage request receiving module 1010, configured to receive a storage request for data in a target object, where the storage request carries attribute information of the target object;
a routing key determining module 1020, configured to determine a routing key of the target object according to the attribute information of the target object;
a fragment number determining module 1030, configured to determine a fragment number of the target object according to the routing key of the target object;
the target storage node determining module 1040 is configured to query a segment tree according to the segment number of the target object, and determine a target storage node of data in the target object;
a data storage module 1050, configured to store data in the target object based on the target storage node;
a fragment tree construction module 1060, configured to construct the fragment tree;
wherein the spanning tree construction module 1060 may include:
the fragment number acquisition sub-module 10601 is configured to acquire the fragment number of the th sample object and the fragment number of the second sample object;
an th identification information determining sub-module 10602, configured to use the last N-bit character of the fragment number of the th sample object as identification information of a th node, where N is a positive integer;
the second identification information determining sub-module 10603 is configured to use the fragment number of the second sample object as identification information of a second node;
the shard tree constructing sub-module 10604 is configured to construct a shard tree based on the identification information of the th node and the identification information of the second node.
In , the target storage node determining module may include:
the post-preset bit character determining submodule is used for determining post-preset bit characters in the fragment number of the target object and taking the post-preset bit characters as current character strings;
the judging submodule is used for judging whether a node matched with the current character string exists in the fragment tree or not;
a current character string determining submodule, configured to increase the length of the post-preset-bit character if a node matching the current character string exists in the segment tree, and use the character after the length is increased as the current character string; repeating the steps: judging whether a node matched with the current character string exists in the fragment tree or not;
a target character determining submodule, configured to determine, if there is no node in the segment tree that matches the current character string, a character of the current character string that is not increased in length for the last times, as a target character;
, a target storage node determining sub-module for determining the node matched by the target character as the target storage node of the data in the target object.
In embodiments, the current string determination submodule may include:
a fragment number judgment unit, configured to judge whether the current character string is the same as the fragment number of the target object if a node matching the current character string exists in the fragment tree;
and the character length increasing unit is used for increasing the length of the post-preset-bit character if the current character string is different from the fragment number of the target object.
In embodiments, the apparatus may further comprise:
and the second target storage node determining submodule is used for determining the node corresponding to the current character string as the target storage node of the data in the target object if the current character string is the same as the fragment number of the target object.
In embodiments, the apparatus may further comprise:
the child node construction module is used for constructing child nodes of the target storage node;
in embodiments, the data storage module may include:
and the data storage submodule is used for storing the data in the target object based on the child nodes of the target storage node.
In embodiments, the apparatus may further comprise:
the preset identification information judgment module is used for judging whether the identification information of the target object is preset identification information or not;
in , the child construction module may include:
th child node constructing unit, configured to construct a child node of the target storage node when the identification information of the target object is preset identification information.
In embodiments, the apparatus may further comprise:
the residual space judging module is used for judging whether the residual space of the target storage node is smaller than a preset threshold value or not;
in , the child construction module may include:
and the second child node constructing unit is used for constructing the child node of the target storage node when the residual space of the target storage node is smaller than a preset threshold value.
The data storage device of the application can be applied to a data access module of a notice system, and the data access module supports virtual partitions, the new variable-length suffix matching fragment and the precise matching fragment further enrich the flexible mechanism of the data storage fragment, and particularly realize the fragment of a specific user, such as the separate fragmentation and isolation of data of active clients or VIP large clients.
The device and method embodiments in the device embodiment described are based on the same inventive concept.
The embodiment of the present application provides data storage devices, which include a processor and a memory, where the memory stores at least instructions, at least segments of programs, code sets or instruction sets, and the at least instructions, the at least segments of programs, the code sets or instruction sets are loaded and executed by the processor to implement the data storage method provided by the above method embodiment.
The data storage device of the application can be applied to a data access module of a notice system, and the data access module supports virtual partitions, the new variable-length suffix matching fragment and the precise matching fragment further enrich the flexible mechanism of the data storage fragment, and particularly realize the fragment of a specific user, such as the separate fragmentation and isolation of data of active clients or VIP large clients.
Embodiments of the present application further provide computer-readable storage media that can be disposed in a terminal to store at least instructions, at least program segments, code sets, or instruction sets for implementing the data storage method in the method embodiments, wherein the at least instructions, the at least program segments, the code sets, or the instruction sets are loaded and executed by the processor to implement the data storage method provided by the above method embodiments.
The computer readable storage medium of the present application may be applied in a data access module of a notification system, the data access module supporting virtual partitioning, new variable length suffix matching shards and exact matching shards further enrich the flexible mechanism of data storage shards.
Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.
The memory according to the embodiments of the present specification may be used to store software programs and modules, and the processor may execute various functional applications and data processing by operating the software programs and modules stored in the memory, and the memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, application programs required for functions, and the like, and the data storage area may store data created according to the use of the device, and the like.
For example, fig. 11 is a hardware structure block diagram of a server of data storage methods provided in the embodiments of the present application, as shown in fig. 11, the server 1100 may have relatively large differences due to different configurations or performances, and may include or or more Central Processing Units (CPUs) 1110 (the processors 1110 may include but are not limited to Processing devices such as a microprocessor MCU or a programmable logic device FPGA), a memory 1130, or or more storage media 1120 storing application programs 1123 or data 1122 (for example, or or more mass storage devices) for storing data, wherein the memory 1130 and the storage media may be transient storage or persistent storage media, the programs stored in the storage media 1120 may include or or more modules, each module may include an instruction for operating a corresponding server in a series 587 or 587, a series of servers, and may further include an instruction for executing a corresponding operation on a wireless network, such as Windows system running on a wireless network, a Windows system, a wireless network.
The I/O Interface 1140 may be used to receive or transmit data via networks, examples of which may include a wireless Network provided by a communication provider of the server 1100. in examples, the I/O Interface 1140 may include Network adapters (NICs) that may be connected to other Network devices via a base station to communicate with the Internet. in examples, the I/O Interface 1140 may be a Radio Frequency (RF) module that may be used to communicate with the Internet via wireless.
It will be understood by those skilled in the art that the structure shown in fig. 11 is only an illustration and is not intended to limit the structure of the electronic device. For example, server 1100 may also include more or fewer components than shown in FIG. 11, or have a different configuration than shown in FIG. 11.
As can be seen from the embodiments of the data storage method, the apparatus, the server, or the storage medium provided by the present application, a fragmentation tree is constructed in the present application, which can implement suffix matching and precise matching of node identification information, a routing key of a target object is determined according to attribute information of the target object, and a fragmentation number of the target object is further determined , and a target storage node is determined by querying the fragmentation tree according to the fragmentation number, so that fragmentation storage of data is implemented, and data of an important user can be separately fragmented for storage, thereby increasing a data reading speed and facilitating a user to quickly query data.
It should be noted that the above-mentioned order of the embodiments of the present application is merely for description and does not represent the superiority or inferiority of the embodiments, and that the above-mentioned description describes certain embodiments of the present application other embodiments are within the scope of the appended claims, in cases, the actions or steps recited in the claims may be performed in an order different than the order of the embodiments and still achieve the desired results.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus, device, and storage medium embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in computer readable storage media, which may be read only memory, magnetic or optical disk, etc.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (10)
1, A method of data storage, the method comprising:
receiving a storage request of data in a target object, wherein the storage request carries attribute information of the target object;
determining a routing key of the target object according to the attribute information of the target object;
determining the fragment number of the target object according to the routing key of the target object;
inquiring a fragment tree according to the fragment number of the target object, and determining a target storage node of data in the target object;
storing data in the target object based on the target storage node;
the construction method of the fragmentation tree comprises the following steps:
, the fragment number of the sample object and the fragment number of the second sample object are obtained;
taking the last N-bit character of the slicing number of the -th sample object as identification information of a -th node, wherein N is a positive integer;
taking the fragment number of the second sample object as identification information of a second node;
and constructing the sharding tree based on the identification information of the th node and the identification information of the second node.
2. The method of claim 1, wherein the querying the sharded tree according to the shard number of the target object to determine the target storage node of the data in the target object comprises:
determining a post-preset bit character in the fragment number of the target object, and taking the post-preset bit character as a current character string;
judging whether a node matched with the current character string exists in the fragment tree or not;
if the segment tree has a node matched with the current character string, increasing the length of the post-preset character, and taking the character with the increased length as the current character string; repeating the steps: judging whether a node matched with the current character string exists in the fragment tree or not;
if the fragment tree does not have a node matched with the current character string, determining the character of the current character string before the length is increased for the last times as a target character;
and determining the node matched with the target character as a target storage node of the data in the target object.
3. The method of claim 2, wherein if a node matching the current string exists in the tile tree, increasing the length of the post-default bit character comprises:
if the fragment tree has a node matched with the current character string, judging whether the fragment numbers of the current character string and the target object are the same;
if not, increasing the length of the post-preset bit character.
4. The method according to claim 3, wherein after the step of determining whether the current character string is the same as the fragment number of the target object, the method further comprises:
and if the current character string is the same as the fragment number of the target object, determining the node corresponding to the current character string as a target storage node of the data in the target object.
5. The method of claim 1, wherein the step of storing the data in the target object based on the target storage node is preceded by the method further comprising:
constructing child nodes of the target storage node;
correspondingly, the storing the data in the target object based on the target storage node includes:
and storing the data in the target object based on the child nodes of the target storage node.
6. The method of claim 5, wherein prior to the step of constructing child nodes of the target storage node, the method further comprises:
judging whether the identification information of the target object is preset identification information or not;
correspondingly, the constructing the child nodes of the target storage node comprises:
and when the identification information of the target object is preset identification information, constructing child nodes of the target storage node.
7. The method of claim 5, wherein prior to the step of constructing child nodes of the target storage node, the method further comprises:
judging whether the residual space of the target storage node is smaller than a preset threshold value or not;
correspondingly, the constructing the child nodes of the target storage node comprises:
and when the residual space of the target storage node is smaller than a preset threshold value, constructing child nodes of the target storage node.
8, A data storage device, characterized in that the device comprises:
a storage request receiving module, configured to receive a storage request for data in a target object, where the storage request carries attribute information of the target object;
a routing key determining module, configured to determine a routing key of the target object according to the attribute information of the target object;
the fragment number determining module is used for determining the fragment number of the target object according to the routing key of the target object;
the target storage node determining module is used for inquiring the fragment tree according to the fragment number of the target object and determining the target storage node of the data in the target object;
the data storage module is used for storing the data in the target object based on the target storage node;
the fragment number acquisition module is used for acquiring the fragment number of the th sample object and the fragment number of the second sample object;
an th identification information determining module, configured to use the last N-bit character of the slice number of the th sample object as identification information of a th node, where N is a positive integer;
a second identification information determining module, configured to use the fragment number of the second sample object as identification information of a second node;
and the fragmentation tree construction module is used for constructing a fragmentation tree based on the identification information of the th node and the identification information of the second node.
A data storage device of kind, characterized in that, the device comprises a processor and a memory, in which at least instructions, at least program segments, code sets or instruction sets are stored, said at least instructions, said at least program segments, code sets or instruction sets being loaded and executed by said processor to implement the data storage method of any of claims 1-7 and .
10, computer readable storage medium having stored therein at least instructions, at least program segments, a set of codes, or a set of instructions, said at least instructions, said at least program segments, said set of codes, or set of instructions being loaded and executed by a processor to implement the data storage method of any of claims 1-7 and .
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910979709.3A CN110737663A (en) | 2019-10-15 | 2019-10-15 | data storage method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910979709.3A CN110737663A (en) | 2019-10-15 | 2019-10-15 | data storage method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110737663A true CN110737663A (en) | 2020-01-31 |
Family
ID=69269023
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910979709.3A Pending CN110737663A (en) | 2019-10-15 | 2019-10-15 | data storage method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737663A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111523001A (en) * | 2020-04-23 | 2020-08-11 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for storing data |
| CN111600799A (en) * | 2020-05-20 | 2020-08-28 | 金蝶蝶金云计算有限公司 | Fragment routing method, server and computer storage medium |
| CN112069210A (en) * | 2020-08-21 | 2020-12-11 | 北京首汽智行科技有限公司 | Saas platform multi-tenant data isolation method |
| CN112084250A (en) * | 2020-09-15 | 2020-12-15 | 深圳市宝能投资集团有限公司 | Data storage method, data query method and electronic equipment |
| CN115883508A (en) * | 2021-09-26 | 2023-03-31 | 中移物联网有限公司 | Number processing method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1585548A (en) * | 2004-05-25 | 2005-02-23 | 中兴通讯股份有限公司 | Realizing method for distributed access positioning register VLR |
| CN102646133A (en) * | 2012-03-28 | 2012-08-22 | 电子科技大学 | Two-dimensional table stored based on metadata and organization method thereof |
| CN104040539A (en) * | 2012-12-31 | 2014-09-10 | 华为技术有限公司 | Data storage method and apparatus, data operation method and system, and access server |
| CN109284073A (en) * | 2018-09-30 | 2019-01-29 | 北京金山云网络技术有限公司 | Date storage method, device, system, server, control node and medium |
-
2019
- 2019-10-15 CN CN201910979709.3A patent/CN110737663A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1585548A (en) * | 2004-05-25 | 2005-02-23 | 中兴通讯股份有限公司 | Realizing method for distributed access positioning register VLR |
| CN102646133A (en) * | 2012-03-28 | 2012-08-22 | 电子科技大学 | Two-dimensional table stored based on metadata and organization method thereof |
| CN104040539A (en) * | 2012-12-31 | 2014-09-10 | 华为技术有限公司 | Data storage method and apparatus, data operation method and system, and access server |
| CN109284073A (en) * | 2018-09-30 | 2019-01-29 | 北京金山云网络技术有限公司 | Date storage method, device, system, server, control node and medium |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111523001A (en) * | 2020-04-23 | 2020-08-11 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for storing data |
| CN111523001B (en) * | 2020-04-23 | 2023-03-14 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for storing data |
| CN111600799A (en) * | 2020-05-20 | 2020-08-28 | 金蝶蝶金云计算有限公司 | Fragment routing method, server and computer storage medium |
| CN111600799B (en) * | 2020-05-20 | 2022-05-20 | 金蝶蝶金云计算有限公司 | Fragment routing method, server and computer storage medium |
| CN112069210A (en) * | 2020-08-21 | 2020-12-11 | 北京首汽智行科技有限公司 | Saas platform multi-tenant data isolation method |
| CN112084250A (en) * | 2020-09-15 | 2020-12-15 | 深圳市宝能投资集团有限公司 | Data storage method, data query method and electronic equipment |
| CN115883508A (en) * | 2021-09-26 | 2023-03-31 | 中移物联网有限公司 | Number processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110737663A (en) | data storage method, device, equipment and storage medium | |
| US10459899B1 (en) | Splitting database partitions | |
| US10467245B2 (en) | System and methods for mapping and searching objects in multidimensional space | |
| CN110489059B (en) | Data cluster storage method and device and computer equipment | |
| JP6626211B2 (en) | Method and apparatus for processing short link and short link server | |
| WO2019200714A1 (en) | Server connection method, computer readable storage medium, terminal device, and apparatus | |
| JP4671332B2 (en) | File server that converts user identification information | |
| US11704320B2 (en) | Processing queries using an index generated based on data segments | |
| US10073876B2 (en) | Bloom filter index for device discovery | |
| JP5759881B2 (en) | Information processing system | |
| TWI734744B (en) | Method, device and system for synchronizing routing table | |
| CN102246172A (en) | System and method for distributed index searching of electronic content | |
| CN111723073B (en) | Data storage processing method, device, processing system and storage medium | |
| CN111611225A (en) | Data storage management method, query method, device, electronic equipment and medium | |
| US20070168364A1 (en) | Non-Mutating Tree-Structured File Identifiers | |
| CN110727738B (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| US8015195B2 (en) | Modifying entry names in directory server | |
| JP4718205B2 (en) | Selective Web information collection device | |
| CN115918110A (en) | Spatial search using key-value store | |
| CN114415971B (en) | Data processing method and device | |
| KR102503958B1 (en) | Apparatus for placing query in distributed networks environment and method thereof | |
| JP6233846B2 (en) | Variable-length nonce generation | |
| JP7390356B2 (en) | Identifying records for tenant identifier conversion after cloning | |
| CN112003736A (en) | Multi-node configuration information determining method, device and storage medium | |
| Abawajy et al. | A framework for scalable distributed provenance storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40021898 Country of ref document: HK |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |