CN111147226B - Data storage method, device and storage medium - Google Patents
Data storage method, device and storage medium Download PDFInfo
- Publication number
- CN111147226B CN111147226B CN201811303561.3A CN201811303561A CN111147226B CN 111147226 B CN111147226 B CN 111147226B CN 201811303561 A CN201811303561 A CN 201811303561A CN 111147226 B CN111147226 B CN 111147226B
- Authority
- CN
- China
- Prior art keywords
- storage
- data
- target
- hash
- target storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
- H04L9/0643—Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/901—Buffering arrangements using storage descriptor, e.g. read or write pointers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Power Engineering (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method, a data storage device and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: receiving a data storage instruction, wherein the data storage instruction carries a preset key value corresponding to data to be stored; obtaining target hash ring information corresponding to the latest version number from hash ring information corresponding to the latest version number, wherein the hash ring information is synchronously obtained from storage management nodes, and when the number of the storage nodes is monitored to change by the storage management nodes, the hash ring information corresponding to each version number is determined by a preset hash algorithm based on the address information of the changed storage nodes; and determining a first target storage node based on the preset key value and the target hash ring information, and storing the data into the first target storage node. The embodiment of the invention can avoid the need of data migration in the data storage process, improves the storage performance of the system and improves the safety of data storage.
Description
Technical Field
The embodiment of the invention relates to the technical field of data storage, in particular to a data storage method, a data storage device and a storage medium.
Background
In some application scenarios, the storage system may implement data storage based on a consistent hash algorithm, where the implementation includes: the storage management node in the storage system constructs an abstract closed hash ring, a plurality of hash values are uniformly distributed on the hash ring, the corresponding hash value is determined through a hash algorithm based on the address information of the storage node in the storage system, the position of the hash value is searched on the hash ring, and the storage node is configured to the position, so that data storage is realized based on the storage node on the hash ring.
In some embodiments, the number of storage nodes in the storage system may vary, for example, as the amount of data grows in large scale, it may be desirable to increase the number of storage nodes in the storage system. For a storage system for realizing data storage based on a consistent hash algorithm, when the number of storage nodes changes, in order to ensure that the stored data can be successfully acquired later, data migration may be required, for example, the data of a certain original storage node on a hash ring may need to be migrated to a newly added storage node.
However, in the above implementation manner, if the data size is very large, performing the data migration operation may affect the system performance, and a data loss phenomenon may occur during the data migration process, which results in poor security of data storage.
Disclosure of Invention
The embodiment of the invention provides a data storage method, a data storage device and a storage medium, which can solve the problem that data migration is required when the number of storage nodes in a storage system changes. The technical scheme is as follows:
in a first aspect, there is provided a data storage method, the method comprising:
receiving a data storage instruction, wherein the data storage instruction carries a preset key value corresponding to data to be stored;
obtaining target hash ring information corresponding to the latest version number from hash ring information corresponding to a plurality of version numbers, wherein the hash ring information is synchronously obtained from storage management nodes, and when the storage management nodes monitor that the number of the storage nodes changes, the hash ring information corresponding to each version number is determined by a preset hash algorithm based on the address information of the changed storage nodes;
and determining a first target storage node based on the preset key value and the target hash ring information, and storing the data into the first target storage node.
Optionally, after the storing the data in the first target storage node, the method further includes:
receiving a data acquisition instruction, wherein the data acquisition instruction carries the preset key value;
Determining a corresponding hash value through the preset hash algorithm based on the preset key value;
and determining a plurality of second target storage nodes based on the hash values and the hash ring information, and acquiring the data from the second target storage nodes.
Optionally, when the hash ring information corresponding to each version number includes a hash value interval, determining, based on the hash value and the hash ring information, a plurality of second target storage nodes includes:
for each hash ring information in the plurality of hash ring information, determining a target hash value interval in which the hash value is located in each hash ring information;
searching the nearest storage node in the clockwise direction in each target hash value interval on the hash ring, determining the searched storage node as the second target storage node, and obtaining the plurality of second target storage nodes.
Optionally, the acquiring the data from the plurality of second target storage nodes includes:
sorting the plurality of second target storage nodes according to the order of version numbers from new to old;
sending a data acquisition request to a first second target storage node in the sorted plurality of second target storage nodes, and stopping sending the data acquisition request when the first second target storage node returns the data; and when the first second target storage node returns an acquisition failure response, sending the data acquisition request to the next second target storage node in the ordered plurality of second target storage nodes until the data is acquired.
Optionally, before the ordering of the plurality of second target storage nodes according to the order of version numbers from new to old, the method further includes:
when the plurality of second target storage nodes comprise the second target storage nodes with the same version numbers, removing the second target storage nodes with the same version numbers from the plurality of second target storage nodes;
accordingly, the sorting the plurality of second target storage nodes according to the order of the version numbers from new to old includes:
and sequencing the rest second target storage nodes according to the order of the version numbers from new to old.
Optionally, before obtaining the target hash ring information corresponding to the latest version number from the hash ring information corresponding to the version numbers, the method further includes:
sending an information synchronization request to the storage management node every preset time length, wherein the information synchronization request is used for requesting the storage management node to return hash ring information corresponding to the version numbers;
and receiving and storing hash ring information corresponding to the version numbers sent by the storage management node.
Optionally, the information synchronization request is further configured to request the storage management node to return address information of each storage node, and the receiving and storing hash ring information corresponding to a plurality of version numbers sent by the storage management node includes:
Receiving and storing hash ring information corresponding to a plurality of version numbers sent by the storage management node and address information of each storage node;
accordingly, the storing the data in the first target storage node includes:
and storing the data into the first target storage node based on the address information of the first target storage node.
In a second aspect, there is provided a data storage device, the device comprising:
the first receiving module is used for receiving a data storage instruction, wherein the data storage instruction carries a preset key value corresponding to data to be stored;
the acquisition module is used for acquiring target hash ring information corresponding to the latest version number from hash ring information corresponding to the version numbers, wherein the hash ring information is synchronously obtained from storage management nodes, and when the storage management nodes monitor that the number of the storage nodes changes, the hash ring information corresponding to each version number is determined and obtained through a preset hash algorithm based on the address information of the changed storage nodes;
and the storage module is used for determining a first target storage node based on the preset key value and the target hash ring information and storing the data into the first target storage node.
Optionally, the apparatus further comprises:
the second receiving module is used for receiving a data acquisition instruction, wherein the data acquisition instruction carries the preset key value;
the determining module is used for determining a corresponding hash value through the preset hash algorithm based on the preset key value;
the determining module is further configured to determine a plurality of second target storage nodes based on the hash value and the hash ring information, and obtain the data from the plurality of second target storage nodes.
Optionally, the determining module is configured to:
when the hash ring information corresponding to each version number comprises a hash value interval, determining a target hash value interval in which the hash value is located in each hash ring information for each hash ring information in the plurality of hash ring information;
searching the nearest storage node in the clockwise direction in each target hash value interval on the hash ring, determining the searched storage node as the second target storage node, and obtaining the plurality of second target storage nodes.
Optionally, the determining module is configured to:
sorting the plurality of second target storage nodes according to the order of version numbers from new to old;
Sending a data acquisition request to a first second target storage node in the sorted plurality of second target storage nodes, and stopping sending the data acquisition request when the first second target storage node returns the data; and when the first second target storage node returns an acquisition failure response, sending the data acquisition request to the next second target storage node in the ordered plurality of second target storage nodes until the data is acquired.
Optionally, the determining module is configured to:
when the plurality of second target storage nodes comprise the second target storage nodes with the same version numbers, removing the second target storage nodes with the same version numbers from the plurality of second target storage nodes;
and sequencing the rest second target storage nodes according to the order of the version numbers from new to old.
Optionally, the apparatus further comprises:
the synchronization module is used for sending an information synchronization request to the storage management node every preset time length, wherein the information synchronization request is used for requesting the storage management node to return hash ring information corresponding to the version numbers;
and the receiving and storing module is used for receiving and storing hash ring information corresponding to the version numbers sent by the storage management node.
Optionally, the receiving and storing module is further configured to: receiving and storing hash ring information corresponding to a plurality of version numbers sent by the storage management node and address information of each storage node;
the storage module is used for storing the data into the first target storage node based on the address information of the first target storage node.
In a third aspect, there is provided a computer readable storage medium having stored thereon instructions which, when executed by a processor, implement the data storage method of the first aspect described above.
In a fourth aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the data storage method of the first aspect described above.
The technical scheme provided by the embodiment of the invention has the beneficial effects that:
and receiving a data storage instruction carrying a preset key value corresponding to data, and acquiring target hash ring information corresponding to the latest version number from the hash ring information corresponding to the version numbers. The hash ring information corresponding to the version numbers is obtained synchronously from the storage management node, and the hash ring information corresponding to each version number is generated by the storage management node based on the address information of the changed storage node after detecting that the number of the storage nodes is changed, that is, the target hash ring information corresponding to the latest version number is the information of the latest hash ring, so that a first target storage node is determined based on a preset key value and the target hash ring information when data is stored, and the data is stored in the first target storage node, and therefore, when the data is read later, the hash ring information corresponding to the version numbers synchronously obtained from the storage management node can be read successfully. That is, the embodiment of the invention can avoid data migration in the data storage process, thereby improving the system storage performance and the security of data storage.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a hash ring, according to an illustrative embodiment;
FIG. 2 is a schematic diagram of an implementation environment, shown in accordance with an exemplary embodiment;
FIG. 3 is a flowchart illustrating a method of data storage according to an exemplary embodiment;
FIG. 4 is a schematic diagram of a hash ring, according to an illustrative embodiment;
FIG. 5 is a flow chart illustrating one type of data storage or reading according to another exemplary embodiment;
FIG. 6 is a schematic diagram of a data storage device according to an exemplary embodiment;
FIG. 7 is a schematic diagram of a data storage device according to another exemplary embodiment;
FIG. 8 is a schematic diagram of a data storage device according to another exemplary embodiment;
Fig. 9 is a schematic diagram of a computer device 800, according to an example embodiment.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
Before describing the data storage method provided by the embodiment of the present invention in detail, the nouns, application scenarios and implementation environments related to the embodiment of the present invention are first described briefly.
First, the terms related to the embodiments of the present invention will be briefly described.
Expansion: in a distributed storage system, a case where the number of storage nodes increases is called capacity expansion.
Shrinking volume: in a distributed storage system, a case where the number of storage nodes is reduced is called a capacity reduction. For example, when a storage node in a storage system is not on-line for a long time, the storage node is automatically placed off-line, which is called a capacitor.
Secondly, the application scene related to the embodiment of the invention is simply introduced.
During data storage, the number of storage nodes in the storage system may change, for example, in a capacity expansion scenario, the number of storage nodes increases, and for example, in a capacity reduction scenario, the number of storage nodes decreases. For a storage system that implements data storage based on a consistent hash algorithm, when the number of storage nodes changes, data migration may be required when storing data in order to ensure that the data may be successfully read later. For example, referring to fig. 1, fig. 1 is a schematic structural diagram of a hash ring according to an exemplary embodiment, where the hash ring originally includes a storage node a, a storage node B, and a storage node C, and a hash value corresponding to certain data stored on the storage node C is located at a position 1 of the hash ring. Since the process of data reading generally includes: determining the position of the hash value of the data on the hash ring, and finding the first storage node from the position along the clockwise direction to be the storage position of the data. Therefore, when the storage node D is added and the storage node D is located as shown in the figure, the data needs to be migrated from the storage node C to the storage node D, so that the subsequent successful acquisition of the hash value based on the data can be ensured to read the data.
Next, an implementation environment related to the embodiment of the present invention will be briefly described.
Referring to fig. 2, fig. 2 is a schematic diagram illustrating an implementation environment according to an exemplary embodiment. The implementation environment mainly comprises a client 210 and a storage system 220, wherein the storage system 220 comprises a storage management node 220a and a storage node 220b. The client 210 may establish a connection with the storage management node 220a and the storage node 220b, respectively, via a network.
The client 210 is mainly configured to implement the data storage method provided by the embodiment of the present invention, in one possible implementation manner, the client 210 may be configured with an SDK (Software Development Kit ), and the client 210 may call the SDK to implement the data storage method provided by the embodiment of the present invention, in other words, the operation of the user on the data and the interaction between the client 210 and the storage system 220 may be completed through the SDK, and it should be noted that, all the clients described below refer to the installation of the SDK and the implementation of the related operation through the SDK. Further, the client 210 may be configured in a computer device.
The storage management node 220a is configured to provide a storage service, for example, may be configured to allocate a storage node, or may be configured to determine hash ring information, etc. In one possible implementation, one or more storage nodes may be randomly selected from the storage system as the storage management node 220a.
The storage node 220b is configured to store data, such as files, audio, video, pictures, etc. Further, client 210 may write data to storage node 220b or may download data from storage node 220 b. In implementation, the storage system 220 generally includes a plurality of storage nodes, and further, in the storage system 220 that implements data storage based on a consistent hashing algorithm, the plurality of storage nodes are all configured on a hash ring.
After the nouns, application scenarios and implementation environments related to the embodiments of the present invention are introduced, the data storage method provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Referring to fig. 3, fig. 3 is a flowchart illustrating a data storage method according to an exemplary embodiment, where the data storage method may be applied to the implementation environment shown in fig. 2, and the data storage method may include the following implementation steps:
step 301: and receiving a data storage instruction, wherein the data storage instruction carries a preset key value corresponding to the data to be stored.
The data storage instruction may be triggered by a user, and the user may trigger a preset operation, where the preset operation may include a clicking operation, a sliding operation, and the like, and the embodiment of the present invention is not limited thereto.
In addition, each complete data corresponds to a unique preset key value. The preset key value can be set by a user according to the actual requirement in a self-defined manner, or can be randomly generated by the client, which is not limited by the embodiment of the invention.
For example, the client may be provided with a user interaction interface, the user interaction interface may be provided with an input box and a data storage option, when the user wants to store data, a preset key value corresponding to the data to be stored may be input in the input box, and then the data storage option may be clicked to trigger the data storage instruction, so that the data storage instruction carries the preset key value corresponding to the data to be stored.
It should be noted that, when the preset key value is randomly generated by the client, in one possible implementation manner, the client may display the preset key value to the user, so that the user may learn and record the preset key value, so that the data may be read based on the preset key value later. In another possible implementation manner, the client may also store the preset key value locally, so that the preset key value may be automatically loaded when the data is read later, so that the data corresponding to the preset key value is read based on the preset key value.
Step 302: and acquiring target hash ring information corresponding to the latest version number from hash ring information corresponding to a plurality of version numbers, wherein the hash ring information is synchronously acquired from a storage management node, and when the storage management node monitors that the number of the storage nodes changes, the hash ring information corresponding to each version number is determined by a preset hash algorithm based on the address information of the changed storage node.
The hash ring information comprises at least one hash value interval, and the hash value interval is obtained by dividing the hash ring by a storage node. Typically, the hash values in the hash ring have a spatial range of [0,2 ] 32 ]In this way, the storage node cuts the hash ring into a plurality of intervals. For example, referring to fig. 4, fig. 4 is a schematic diagram illustrating a hash ring according to an exemplary embodiment, where the hash ring includes a storage node m, a storage node n, and a storage node q, and the hash ring is cut into three hash value intervals, and the three hash value intervals include a first hash value interval, a second hash value interval, and a third hash value interval.
For the storage management node, the number change condition of the storage nodes in the storage system can be always monitored and managed, so that a plurality of hash ring information can be generated according to the number change condition of the storage nodes. For example, when the capacity of the storage node in the storage system is insufficient, the storage node may be added to the storage system, that is, the user may perform the capacity expansion operation. In implementations, a user may send a node addition request to a storage management node through a user interaction interface in a client, which may, in some embodiments, carry address information of the storage node to be added. After receiving the node addition request, the storage management node records the address information of the storage nodes with changed numbers, and in a possible implementation manner, the address information of the storage nodes with changed numbers can be recorded in a list form. Further, the storage management node may determine a new hash ring information through a preset hash algorithm based on the address information of the storage node after the number change, that is, the number of hash ring information stored in the storage management node is increased.
That is, each time the number of storage nodes in the storage system is monitored to change, the storage management node generates a new hash ring based on the address information of the changed storage nodes, in other words, when the number of storage nodes is not changed, only one hash ring information is stored in the storage system, and when the number of storage nodes is changed, a plurality of hash ring information are stored in the storage system. Further, the storage management node may store the generated plurality of hash ring information in the form of a list.
The preset hash algorithm may be set by a user according to an implementation requirement, or may be set by default by the storage management node, which is not limited in the embodiment of the present invention.
Further, the storage management node may add a version number to the newly generated hash ring information. In an embodiment, the version number is in an increasing trend over time. For example, the version number of the hash ring information corresponding to the original hash ring in the storage system may be set to 1, when the number of storage nodes in the storage system changes for the first time, the storage management node may set the version number corresponding to the hash ring information generated based on the address information of the storage nodes after the number changes to 2, when the number of storage nodes in the storage system changes for the second time, the storage management node may set the version number corresponding to the hash ring information generated based on the address information of the storage nodes after the number changes to 3, and so on.
For a client, hash ring information corresponding to a plurality of version numbers can be synchronized from a storage management node periodically, and in one possible implementation manner, the client sends an information synchronization request to the storage management node every a preset time period, where the information synchronization request is used for requesting the storage management node to return the hash ring information corresponding to the plurality of version numbers, and receives and stores the hash ring information corresponding to the plurality of version numbers sent by the storage management node.
The preset duration may be set by a user in a user-defined manner according to actual requirements, or may be set by default by the client, which is not limited in the embodiment of the present invention.
For example, if the preset duration is one week, and the storage management node is assumed to store hash ring information corresponding to a plurality of version numbers in the form of a list, the client may synchronously store the list of hash ring information corresponding to the plurality of version numbers from the storage management node every one week.
In this way, in the process of storing data, the target hash ring information corresponding to the latest version number can be obtained from the hash ring information corresponding to the plurality of version numbers, namely, the hash ring information determined by the storage management node after the number of the storage nodes in the latest storage system is changed is obtained.
Further, the information synchronization request is further used for requesting the storage management node to return the address information of each storage node, and at this time, the client receives and stores hash ring information corresponding to the version numbers sent by the storage management node and the address information of each storage node.
That is, the storage management node may synchronize address information of each storage node included in the storage system to the client in addition to synchronizing hash ring information corresponding to the plurality of version numbers to the client. Further, the storage management node may synchronize the address information of each storage node to the client in the form of an address information list, i.e. the address information list comprises the address information of each storage node.
Step 303: and determining a first target storage node based on the preset key value and the target hash ring information.
In one possible implementation manner, determining the specific implementation of the first target storage node based on the preset key value and the target hash ring information may include: based on the preset key value, determining a corresponding hash value through the preset hash algorithm, determining a hash value interval in which the hash value is located in the target hash ring information, searching the nearest storage node on the hash ring in the clockwise direction, and determining the searched storage node as the first target storage node.
For example, referring to fig. 4, assuming that the corresponding hash value is determined to be within the first hash value interval by the above-mentioned preset hash algorithm based on the preset key value, the nearest storage node m can be found on the hash ring in the clockwise direction, and at this time, the storage node m is determined to be the first target storage node.
Step 304: the data is stored in the first target storage node.
In one possible implementation, the client may send a data storage request to the first target storage node, where the data storage request carries data to be stored. After the first target storage node receives the data storage request, the data storage request is analyzed to extract data to be stored from the data storage request, and the data is stored.
Further, the client may store the data into the first target storage node based on the address information of the first target storage node. That is, the client acquires the address information of the first target storage node from the address information of each storage node that is synchronized, and sends the data storage request to the address information of the first target storage node to store data in the first target storage node.
It should be noted that, the above steps 301 to 304 are used to describe an implementation process of storing data in the storage system, and next, an implementation process of reading data from the storage system will be described, specifically please refer to the following steps 305 to 308.
Step 305: and receiving a data acquisition instruction, wherein the data acquisition instruction carries the preset key value.
The data acquisition instruction may be triggered by a user, and the user may be triggered by the above-mentioned preset operation. For example, in some embodiments, the client may provide a user interaction interface that is provided with an input box, as described above, and may also be provided with data acquisition options. When the user wants to download data from the storage system, the user can input a preset key value set when the data is stored in the input box and click on a data acquisition option to trigger the data acquisition instruction.
Step 306: based on the preset key value, determining a corresponding hash value through the preset hash algorithm.
It should be noted that, the preset hash algorithm used herein is a hash algorithm used when data is stored, that is, the corresponding hash value needs to be determined by the same hash function, so that the position where the hash value is located can be successfully found out from the hash ring.
Step 307: a plurality of second target storage nodes is determined based on the hash value and the plurality of hash ring information.
As described above, the hash ring information corresponding to each version number includes a hash value interval, and determining, based on the hash value and the hash ring information, a specific implementation of the second target storage nodes may include: and for each hash ring information in the plurality of hash ring information, determining a target hash value interval in which the hash value is located in the each hash ring information, searching the nearest storage node in the clockwise direction in each target hash value interval on the hash ring, and determining the searched storage node as the second target storage node to obtain the plurality of second target storage nodes.
That is, the client may determine a second target storage node based on the hash value and the hash ring information corresponding to each version number, and thus, may determine a plurality of second target storage nodes based on the hash value and the hash ring information corresponding to the plurality of version numbers. For example, assuming that there are three hash ring information, the client may determine three second target storage nodes, where each second target storage node corresponds to a unique version number.
Step 308: the data is retrieved from the plurality of second target storage nodes.
In one possible implementation, the specific implementation process of acquiring the data from the plurality of second target storage nodes may include: sequencing the plurality of second target storage nodes according to the sequence of the version numbers from new to old, sending a data acquisition request to a first second target storage node in the sequenced plurality of second target storage nodes, and stopping the operation of sending the data acquisition request when the first second target storage node returns the data; and when the first second target storage node returns an acquisition failure response, sending the data acquisition request to the next second target storage node in the ordered plurality of second target storage nodes until the data is acquired.
Since the data is stored based on the target hash ring information corresponding to the latest version number, in order to enable quick data acquisition, the determined plurality of second target storage nodes are ordered in the order of version numbers from new to old, for example, as described above, the version numbers corresponding to the plurality of hash ring information may present increasing patterns with time, in which case the plurality of second target storage nodes are ordered in the order of version numbers from large to small.
And then, starting from the first second target storage node after sequencing, sending a data acquisition request to the first second target storage node, and if the first second target storage node returns an acquisition failure response to indicate that the data acquisition fails, then sending the data acquisition request to the second target storage node after sequencing until the data is acquired. Otherwise, if the first target storage node and the second target storage node return data, indicating that the data acquisition is successful, stopping the data acquisition operation.
Further, before ordering the plurality of second target storage nodes according to the order of the version numbers from new to old, when the plurality of second target storage nodes include second target storage nodes with the same version numbers, the second target storage nodes with the same version numbers are removed from the plurality of second target storage nodes, and at this time, the remaining plurality of second target storage nodes are ordered according to the order of the version numbers from new to old.
It will be appreciated that among the determined plurality of second target storage nodes, there may be a second storage node of the same version number, and thus, a certain second storage node may be repeatedly read when data is acquired in the above manner. Therefore, in order to reduce the operation burden of the client, before the plurality of second target storage nodes are ordered in the order of version numbers from new to old, a deduplication process may be performed, that is, the second target storage nodes with the same version number are removed from the plurality of second target storage nodes, and then the remaining plurality of second target storage nodes are ordered, so that repeated reading may be avoided.
In addition, in the data reading process, the client may send a data acquisition request to each second target storage node based on the address information of each second target storage node, so as to read data from the second target storage node.
In the embodiment of the invention, a data storage instruction carrying a preset key value corresponding to data is received, and target hash ring information corresponding to the latest version number is obtained from hash ring information corresponding to a plurality of version numbers. The hash ring information corresponding to the version numbers is obtained synchronously from the storage management node, and the hash ring information corresponding to each version number is generated by the storage management node based on the address information of the changed storage node after detecting that the number of the storage nodes is changed, that is, the target hash ring information corresponding to the latest version number is the information of the latest hash ring, so that a first target storage node is determined based on a preset key value and the target hash ring information when data is stored, and the data is stored in the first target storage node, and therefore, when the data is read later, the hash ring information corresponding to the version numbers synchronously obtained from the storage management node can be read successfully. That is, the embodiment of the invention can avoid data migration in the data storage process, thereby improving the system storage performance and the security of data storage.
Next, please refer to fig. 5, which is a flow chart illustrating a data storage or reading according to an exemplary embodiment. The data storage and reading process is briefly described herein in connection with fig. 5. Two scenarios are included here:
first scenario: only one hash ring scenario exists in the storage management node.
1. The client periodically synchronizes hash ring information from the storage management node, and accordingly, the storage management node returns the hash ring information. Further, the storage management node may synchronize address information of each storage node included in the storage system to the client.
2. When data need to be uploaded or downloaded, a user can send a data writing instruction or a data acquisition instruction to the client, wherein the data writing instruction or the data acquisition instruction carries a unique preset key value corresponding to the data.
3. The client calculates according to the preset key value and the synchronous hash ring information, and determines a target storage node (a first target storage node or a second target storage node).
4. The client sends a data write request or a data acquisition request to the target storage node. Further, the client sends a data writing request or a data acquisition request to the target storage node based on the address information of the target storage node.
5. The target storage node stores the data or returns the data to be downloaded to the client.
The second scenario: there are multiple hash ring scenarios in the storage management node.
1. The client periodically synchronizes hash ring information from the storage management node, and accordingly, the storage management node returns the hash ring information. Further, the storage management node may synchronize address information of each storage node included in the storage system to the client.
2. When the data is required to be uploaded, a user can send a data writing instruction to the client, wherein the data writing instruction carries a unique preset key value corresponding to the data.
3. And the client calculates according to the preset key value and target hash ring information corresponding to the latest version number in the synchronous hash ring information, and determines a target storage node (a first target storage node).
4. The client sends a data write request to the target storage node. Further, the client sends a data write request to the target storage node based on the address information of the target storage node.
5. The target storage node stores data.
6. When the data needs to be downloaded, the user can send a data acquisition instruction to the client, wherein the data acquisition instruction carries a unique preset key value corresponding to the data.
7. The client calculates according to the preset key value and the synchronous hash ring information, and determines a target storage node (comprising a plurality of second target storage nodes).
8. The client sends a data acquisition request to the target storage node, and further, the client sends the data acquisition request to the target storage node based on the address information of the target storage node. The client judges whether the downloading is successful, and stops the operation when the downloading is successful, otherwise, the next step 9 is entered.
9. The client sends a data acquisition request to the next target storage node until the data download is successful.
Fig. 6 is a schematic diagram of a data storage device that may be implemented in software, hardware, or a combination of both, according to an example embodiment. The data storage device may include:
the first receiving module 610 is configured to receive a data storage instruction, where the data storage instruction carries a preset key value corresponding to data to be stored;
the obtaining module 620 is configured to obtain target hash ring information corresponding to the latest version number from hash ring information corresponding to a plurality of version numbers, where the hash ring information is obtained from storage management nodes synchronously, and when the storage management nodes monitor that the number of storage nodes changes, the hash ring information corresponding to each version number is determined and obtained by a preset hash algorithm based on address information of the changed storage nodes;
The storage module 630 is configured to determine a first target storage node based on the preset key value and the target hash ring information, and store the data into the first target storage node.
Optionally, referring to fig. 7, the apparatus further includes:
the second receiving module 640 is configured to receive a data acquisition instruction, where the data acquisition instruction carries the preset key value;
a determining module 650, configured to determine, based on the preset key value, a corresponding hash value by using the preset hash algorithm;
the determining module 650 is further configured to determine a plurality of second target storage nodes based on the hash value and the hash ring information, and obtain the data from the plurality of second target storage nodes.
Optionally, the determining module 650 is configured to:
when the hash ring information corresponding to each version number comprises a hash value interval, determining a target hash value interval in which the hash value is located in each hash ring information for each hash ring information in the plurality of hash ring information;
searching the nearest storage node in the clockwise direction in each target hash value interval on the hash ring, determining the searched storage node as the second target storage node, and obtaining the plurality of second target storage nodes.
Optionally, the determining module 650 is configured to:
sorting the plurality of second target storage nodes according to the order of version numbers from new to old;
sending a data acquisition request to a first second target storage node in the sorted plurality of second target storage nodes, and stopping sending the data acquisition request when the first second target storage node returns the data; and when the first second target storage node returns an acquisition failure response, sending the data acquisition request to the next second target storage node in the ordered plurality of second target storage nodes until the data is acquired.
Optionally, the determining module 650 is configured to:
when the plurality of second target storage nodes comprise the second target storage nodes with the same version numbers, removing the second target storage nodes with the same version numbers from the plurality of second target storage nodes;
and sequencing the rest second target storage nodes according to the order of the version numbers from new to old.
Optionally, referring to fig. 8, the apparatus further includes:
a synchronization module 660, configured to send an information synchronization request to the storage management node every a preset duration, where the information synchronization request is used to request the storage management node to return hash ring information corresponding to the multiple version numbers;
And the receiving and storing module 670 is configured to receive and store hash ring information corresponding to the version numbers sent by the storage management node.
Optionally, the receiving storage module 670 is further configured to: receiving and storing hash ring information corresponding to a plurality of version numbers sent by the storage management node and address information of each storage node;
the storage module 630 is configured to store the data into the first target storage node based on address information of the first target storage node.
In the embodiment of the invention, a data storage instruction carrying a preset key value corresponding to data is received, and target hash ring information corresponding to the latest version number is obtained from hash ring information corresponding to a plurality of version numbers. The hash ring information corresponding to the version numbers is obtained synchronously from the storage management node, and the hash ring information corresponding to each version number is generated by the storage management node based on the address information of the changed storage node after detecting that the number of the storage nodes is changed, that is, the target hash ring information corresponding to the latest version number is the information of the latest hash ring, so that a first target storage node is determined based on a preset key value and the target hash ring information when data is stored, and the data is stored in the first target storage node, and therefore, when the data is read later, the hash ring information corresponding to the version numbers synchronously obtained from the storage management node can be read successfully. That is, the embodiment of the invention can avoid data migration in the data storage process, improve the system storage performance and improve the security of data storage
It should be noted that: in the data storage device provided in the above embodiment, when implementing the data storage method, only the division of the above functional modules is used for illustration, in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the data storage device and the data storage method provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the data storage device and the data storage method are detailed in the method embodiments and are not repeated herein.
Fig. 9 is a schematic diagram of a computer device 800, according to an example embodiment. The computer device may be equipped with the above-described client, in particular:
the computer device 800 includes a Central Processing Unit (CPU) 801, a system memory 804 including a Random Access Memory (RAM) 802 and a Read Only Memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the central processing unit 801. Computer device 800 also includes a basic input/output system (I/O system) 806 that facilitates the transfer of information between various devices within the computer, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.
The basic input/output system 806 includes a display 808 for displaying information and an input device 809, such as a mouse, keyboard, or the like, for user input of information. Wherein both the display 808 and the input device 809 are connected to the central processing unit 801 via an input output controller 810 connected to the system bus 805. The basic input/output system 806 may also include an input/output controller 810 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input output controller 810 also provides output to a display screen, a printer, or other type of output device.
The mass storage device 807 is connected to the central processing unit 801 through a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable media provide non-volatile storage for the computer device 800. That is, the mass storage device 807 may include a computer readable medium (not shown) such as a hard disk or CD-ROM drive.
Computer readable media may include computer storage media and communication media without loss of generality. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the ones described above. The system memory 804 and mass storage device 807 described above may be collectively referred to as memory.
According to various embodiments of the present application, the computer device 800 may also operate by being connected to a remote computer on a network, such as the Internet. I.e., computer device 800 may be connected to a network 812 through a network interface unit 811 connected to system bus 805, or other types of networks or remote computer systems (not shown) may also be connected to using network interface unit 811.
The memory also includes one or more programs, one or more programs stored in the memory and configured to be executed by the CPU. The one or more programs include data storage methods for performing the embodiments provided herein.
Embodiments of the present application also provide a non-transitory computer-readable storage medium, which when executed by a processor of a computer device, enables the computer device to perform the above-described data storage method.
Embodiments of the present application also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the data storage method described above.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.
Claims (13)
1. A data storage method, characterized by being applied to a client, the method comprising:
receiving a data storage instruction, wherein the data storage instruction carries a preset key value corresponding to data to be stored;
obtaining target hash ring information corresponding to the latest version number from hash ring information corresponding to a plurality of version numbers, wherein the hash ring information is synchronously obtained from storage management nodes, and when the storage management nodes monitor that the number of the storage nodes changes, the hash ring information corresponding to each version number is determined by a preset hash algorithm based on the address information of the changed storage nodes;
determining a first target storage node based on the preset key value and the target hash ring information, and storing the data into the first target storage node;
receiving a data acquisition instruction, wherein the data acquisition instruction carries the preset key value;
determining a corresponding hash value through the preset hash algorithm based on the preset key value;
And determining a plurality of second target storage nodes based on the hash values and the hash ring information, and acquiring the data from the second target storage nodes.
2. The method of claim 1, wherein when the hash-ring information corresponding to each version number includes a hash value interval, the determining a plurality of second target storage nodes based on the hash values and the plurality of hash-ring information includes:
for each hash ring information in the plurality of hash ring information, determining a target hash value interval in which the hash value is located in each hash ring information;
searching the nearest storage node in the clockwise direction in each target hash value interval on the hash ring, determining the searched storage node as the second target storage node, and obtaining the plurality of second target storage nodes.
3. The method of claim 2, wherein the obtaining the data from the plurality of second target storage nodes comprises:
sorting the plurality of second target storage nodes according to the order of version numbers from new to old;
sending a data acquisition request to a first second target storage node in the sorted plurality of second target storage nodes, and stopping sending the data acquisition request when the first second target storage node returns the data; and when the first second target storage node returns an acquisition failure response, sending the data acquisition request to the next second target storage node in the ordered plurality of second target storage nodes until the data is acquired.
4. The method of claim 3, wherein before ordering the plurality of second target storage nodes in order of version number from new to old, further comprising:
when the plurality of second target storage nodes comprise the second target storage nodes with the same version numbers, removing the second target storage nodes with the same version numbers from the plurality of second target storage nodes;
accordingly, the sorting the plurality of second target storage nodes according to the order of the version numbers from new to old includes:
and sequencing the rest second target storage nodes according to the order of the version numbers from new to old.
5. The method as claimed in any one of claims 1 to 4, wherein before obtaining the target hash ring information corresponding to the latest version number from the hash ring information corresponding to the plurality of version numbers, the method further comprises:
sending an information synchronization request to the storage management node every preset time length, wherein the information synchronization request is used for requesting the storage management node to return hash ring information corresponding to the version numbers;
and receiving and storing hash ring information corresponding to the version numbers sent by the storage management node.
6. The method of claim 5, wherein the information synchronization request is further used to request the storage management node to return address information of each storage node, and the receiving and storing hash ring information corresponding to a plurality of version numbers sent by the storage management node includes:
receiving and storing hash ring information corresponding to a plurality of version numbers sent by the storage management node and address information of each storage node;
accordingly, the storing the data in the first target storage node includes:
and storing the data into the first target storage node based on the address information of the first target storage node.
7. A data storage device configured in a client, the device comprising:
the first receiving module is used for receiving a data storage instruction, wherein the data storage instruction carries a preset key value corresponding to data to be stored;
the acquisition module is used for acquiring target hash ring information corresponding to the latest version number from hash ring information corresponding to the version numbers, wherein the hash ring information is synchronously obtained from storage management nodes, and when the storage management nodes monitor that the number of the storage nodes changes, the hash ring information corresponding to each version number is determined and obtained through a preset hash algorithm based on the address information of the changed storage nodes;
The storage module is used for determining a first target storage node based on the preset key value and the target hash ring information and storing the data into the first target storage node;
the second receiving module is used for receiving a data acquisition instruction, wherein the data acquisition instruction carries the preset key value;
the determining module is used for determining a corresponding hash value through the preset hash algorithm based on the preset key value;
the determining module is further configured to determine a plurality of second target storage nodes based on the hash value and the hash ring information, and obtain the data from the plurality of second target storage nodes.
8. The apparatus of claim 7, wherein the determination module is to:
when the hash ring information corresponding to each version number comprises a hash value interval, determining a target hash value interval in which the hash value is located in each hash ring information for each hash ring information in the plurality of hash ring information;
searching the nearest storage node in the clockwise direction in each target hash value interval on the hash ring, determining the searched storage node as the second target storage node, and obtaining the plurality of second target storage nodes.
9. The apparatus of claim 8, wherein the determination module is to:
sorting the plurality of second target storage nodes according to the order of version numbers from new to old;
sending a data acquisition request to a first second target storage node in the sorted plurality of second target storage nodes, and stopping sending the data acquisition request when the first second target storage node returns the data; and when the first second target storage node returns an acquisition failure response, sending the data acquisition request to the next second target storage node in the ordered plurality of second target storage nodes until the data is acquired.
10. The apparatus of claim 9, wherein the determination module is to:
when the plurality of second target storage nodes comprise the second target storage nodes with the same version numbers, removing the second target storage nodes with the same version numbers from the plurality of second target storage nodes;
and sequencing the rest second target storage nodes according to the order of the version numbers from new to old.
11. The apparatus according to any one of claims 7-10, wherein the apparatus further comprises:
The synchronization module is used for sending an information synchronization request to the storage management node every preset time length, wherein the information synchronization request is used for requesting the storage management node to return hash ring information corresponding to the version numbers;
and the receiving and storing module is used for receiving and storing hash ring information corresponding to the version numbers sent by the storage management node.
12. The apparatus of claim 11, wherein the device comprises a plurality of sensors,
the receiving and storing module is further used for: receiving and storing hash ring information corresponding to a plurality of version numbers sent by the storage management node and address information of each storage node;
the storage module is used for storing the data into the first target storage node based on the address information of the first target storage node.
13. A computer readable storage medium having instructions stored thereon, which when executed by a processor, implement the steps of the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811303561.3A CN111147226B (en) | 2018-11-02 | 2018-11-02 | Data storage method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811303561.3A CN111147226B (en) | 2018-11-02 | 2018-11-02 | Data storage method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111147226A CN111147226A (en) | 2020-05-12 |
| CN111147226B true CN111147226B (en) | 2023-07-18 |
Family
ID=70516256
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811303561.3A Active CN111147226B (en) | 2018-11-02 | 2018-11-02 | Data storage method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111147226B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282243B (en) * | 2021-06-09 | 2022-12-02 | 杭州海康威视系统技术有限公司 | Method and device for storing object file |
| CN114844911A (en) * | 2022-04-20 | 2022-08-02 | 网易（杭州）网络有限公司 | Data storage method and device, electronic equipment and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101996217A (en) * | 2009-08-24 | 2011-03-30 | 华为技术有限公司 | Method for storing data and memory device thereof |
| CN102739704A (en) * | 2011-04-02 | 2012-10-17 | 中兴通讯股份有限公司 | Method and system for data migration in peer-to-peer network |
| CN104636286A (en) * | 2015-02-06 | 2015-05-20 | 华为技术有限公司 | Data access method and equipment |
| CN106503139A (en) * | 2016-10-20 | 2017-03-15 | 上海携程商务有限公司 | Dynamic data access method and system |
| CN106951179A (en) * | 2016-01-07 | 2017-07-14 | 杭州海康威视数字技术股份有限公司 | A kind of data migration method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11157517B2 (en) * | 2016-04-18 | 2021-10-26 | Amazon Technologies, Inc. | Versioned hierarchical data structures in a distributed data store |
-
2018
- 2018-11-02 CN CN201811303561.3A patent/CN111147226B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101996217A (en) * | 2009-08-24 | 2011-03-30 | 华为技术有限公司 | Method for storing data and memory device thereof |
| CN102739704A (en) * | 2011-04-02 | 2012-10-17 | 中兴通讯股份有限公司 | Method and system for data migration in peer-to-peer network |
| CN104636286A (en) * | 2015-02-06 | 2015-05-20 | 华为技术有限公司 | Data access method and equipment |
| CN106951179A (en) * | 2016-01-07 | 2017-07-14 | 杭州海康威视数字技术股份有限公司 | A kind of data migration method and device |
| CN106503139A (en) * | 2016-10-20 | 2017-03-15 | 上海携程商务有限公司 | Dynamic data access method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111147226A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104951474B (en) | Method and device for acquiring MySQL binlog incremental log | |
| EP3223165B1 (en) | File processing method, system and server-clustered system for cloud storage | |
| CN107291750B (en) | Data migration method and device | |
| JP6273927B2 (en) | Information processing system, monitoring device, monitoring program, and monitoring method | |
| EP3125501B1 (en) | File synchronization method, server, and terminal | |
| CN107656695B (en) | Data storage and deletion method and device and distributed storage system | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN104348859B (en) | File synchronisation method, device, server, terminal and system | |
| CN111182067A (en) | Data writing method and device based on interplanetary file system IPFS | |
| CN105786539B (en) | File downloading method and device | |
| CN111147226B (en) | Data storage method, device and storage medium | |
| CN112000850B (en) | Method, device, system and equipment for processing data | |
| CN107623705B (en) | Storage mode upgrading method, device and system based on video cloud storage system | |
| CN115004662A (en) | Data synchronization method, data synchronization device, data storage system and computer readable medium | |
| CN110955460A (en) | Service process starting method and device, electronic equipment and storage medium | |
| CN111309693A (en) | Data synchronization method, device and system, electronic equipment and storage medium | |
| CN114422537B (en) | Multi-cloud storage system, multi-cloud data reading and writing method and electronic equipment | |
| JP2016100015A (en) | Synchronization of collaborative document with online document management system | |
| CN115129789A (en) | Bucket index storage method, device and medium of distributed object storage system | |
| CN110677497B (en) | Network medium distribution method and device | |
| CN111966533A (en) | Electronic file management method and device, computer equipment and storage medium | |
| CN106407320B (en) | File processing method, device and system | |
| CN111435342A (en) | Poster updating method, poster updating system and poster management system | |
| CN111209597A (en) | Data verification method and application system | |
| CN111666035B (en) | Management method and device of distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |