CN112511634A - Data acquisition method and device, electronic equipment and storage medium - Google Patents

Data acquisition method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112511634A
CN112511634A CN202011403471.9A CN202011403471A CN112511634A CN 112511634 A CN112511634 A CN 112511634A CN 202011403471 A CN202011403471 A CN 202011403471A CN 112511634 A CN112511634 A CN 112511634A
Authority
CN
China
Prior art keywords
storage node
data
storage
hash value
proxy server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011403471.9A
Other languages
Chinese (zh)
Inventor
谢人超
赵浩然
任语铮
喻嘉艺
黄韬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing University of Posts and Telecommunications
Original Assignee
Beijing University of Posts and Telecommunications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing University of Posts and Telecommunications filed Critical Beijing University of Posts and Telecommunications
Priority to CN202011403471.9A priority Critical patent/CN112511634A/en
Publication of CN112511634A publication Critical patent/CN112511634A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/148—Migration or transfer of sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/566—Grouping or aggregating service requests, e.g. for unified processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/63—Routing a service request depending on the request content or context

Abstract

The embodiment of the invention provides a data acquisition method, a data acquisition device, electronic equipment and a storage medium, wherein the method is applied to a proxy server in a DHT network and comprises the following steps: receiving a first data request sent by a client, wherein the first data request comprises first data information of first data to be requested; determining a first hash value of the first data information according to a preset hash algorithm; determining a first storage node corresponding to the first hash value according to the corresponding relation between the pre-stored hash value and the storage node; sending a first data request to a first storage node, so that the first storage node feeds back an indication message to a proxy server under the condition that first data are migrated to a second storage node; according to the indication message, sending a first data request to a second storage node; and receiving first data fed back by the second storage node according to the first data request, and forwarding the first data to the client.

Description

Data acquisition method and device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of data security technologies, and in particular, to a data acquisition method and apparatus, an electronic device, and a storage medium.
Background
With the continuous evolution of internet technology and the development of production and manufacturing industry, the current internet is gradually changed from 'consumption type' to 'production type', the chinese industry is facing the requirement of digital transformation, and the global manufacturing industry is advancing from the digitalization stage to the networking stage. The industrial internet is a product of deep integration of a new generation of information communication technology and modern industrial technology, is an important carrier for digitalization, networking and intellectualization of manufacturing industry, increasingly becomes a key support of a new industrial revolution and an important foundation stone for deepening 'internet and advanced manufacturing industry', and has all-round, deep and revolutionary influence on future industrial development.
Aiming at massive industrial data generated in a supply chain, the industrial internet can establish the relation among production equipment, raw materials, a control system, product information and people by utilizing an identification analysis system of the industrial internet, is favorable for solving the problems of exchange fusion and public understanding of data inside enterprises and among enterprises, and meets the application requirement of large data in the future industrialized development process. At present, most of the conventional identifier resolution mechanisms are based on a Domain Name System (DNS), that is, centralized storage and management of a single point server. In centralized storage and management of the single-point server, data is all stored in the central server, so that overload of the central server is easily caused, and when the central server fails or is attacked, a large amount of data is easily lost.
Based on this, a data distributed storage method without a central server is provided. The data Distributed storage method is applied to a Distributed Hash Table (DHT) network, and the DHT network is realized by using a Hash algorithm. In a DHT network, each node is responsible for a small range of routing and for storing a small portion of data, i.e. the data is stored in a distributed manner in multiple nodes, thereby enabling addressing and storage of the entire DHT network. The method effectively solves the problems of overweight load, easy loss of a large amount of data and the like in centralized storage and management of the single-point server, and has the characteristics of decentralization, collective maintenance, safety, credibility, tamper resistance and the like.
However, when a node needs to exit the DHT network, the node migrates data stored by itself to an adjacent node, but in the process of migrating the data, the node cannot provide a data identifier resolution service, so that the service continuity of the node is poor.
Disclosure of Invention
An object of the embodiments of the present invention is to provide a data acquisition method, so as to improve continuity of a data identifier resolution service of a node in a DHT network. The specific technical scheme is as follows:
in order to achieve the above object, an embodiment of the present invention provides a data acquisition method, which is applied to a proxy server in a distributed hash table DHT network, where the proxy server is connected to a plurality of storage nodes, and the method includes:
receiving a first data request sent by a client, wherein the first data request comprises first data information of first data to be requested;
determining a first hash value of the first data information according to a preset hash algorithm;
determining a first storage node corresponding to the first hash value according to a corresponding relation between the pre-stored hash value and the storage node;
sending the first data request to the first storage node, so that the first storage node feeds back an indication message to the proxy server when the first data is migrated to a second storage node, wherein the indication message is used for informing the proxy server that the first data is migrated to the second storage node;
sending the first data request to the second storage node according to the indication message;
and receiving the first data fed back by the second storage node according to the first data request, and forwarding the first data to the client.
Optionally, the method further includes:
determining a third storage node according to a corresponding relation between preset storage nodes and backup storage nodes, wherein the number of the backup storage nodes corresponding to the third storage node is smaller than the preset storage number, and the backup storage node corresponding to each storage node is a storage node for backing up original data on the storage node;
determining a fourth storage node which does not store original data on the third storage node and has the largest residual storage space from a plurality of storage nodes connected with the proxy server;
and backing up the original data on the third storage node to the fourth storage node, and storing the corresponding relation between the third storage node and the fourth storage node.
Optionally, after sending the first data request to the first storage node, the method further includes:
if response information or the indication information aiming at the first data request is not received, the first data request is sent to the first storage node again, and the response information comprises the first data;
if a preset number of first data requests are continuously sent and no response information or indication information is received, determining backup storage nodes corresponding to the first storage nodes according to the corresponding relation between preset storage nodes and backup storage nodes, wherein the backup storage node corresponding to each storage node is a storage node for backing up original data on the storage node;
and forwarding the first data request to a backup storage node corresponding to the first storage node.
Optionally, the plurality of storage nodes include a physical storage node and a virtual storage node, and the method further includes:
receiving a data storage request, wherein the data storage request comprises second data information of second data to be stored;
determining a second hash value of the second data information according to a preset hash algorithm;
determining a fifth storage node corresponding to the second hash value according to the corresponding relation between the pre-stored hash value and the storage node;
if the fifth storage node is a virtual storage node, determining a physical storage node corresponding to the fifth storage node according to a pre-stored corresponding relationship between the physical storage node and the virtual storage node;
and storing the second data into a physical storage node corresponding to the fifth storage node.
Optionally, the method further includes:
receiving a data migration request sent by a sixth storage node to be exited, wherein the data migration request comprises third data information of each original data stored by the sixth storage node;
determining a third hash value of third data information of each original data on the sixth storage node according to a preset hash algorithm;
determining a new storage node corresponding to each third hash value according to a correspondence between a pre-stored hash value and a storage node, and using the new storage node as a new storage node corresponding to each original data on the sixth storage node;
sending a data migration instruction to the sixth storage node, where the data migration instruction includes an identifier of a new storage node corresponding to each piece of original data on the sixth storage node, so that the sixth storage node migrates each piece of original data on the sixth storage node to the corresponding new storage node according to the data migration instruction.
Optionally, after the sending the data migration instruction to the sixth storage node, the method further includes:
receiving a migration end message sent by the sixth storage node, where the migration end message indicates that migration of each original data on the sixth storage node is completed;
and deleting the corresponding relation between each third hash value and the sixth storage node according to the migration ending message, and storing the corresponding relation between each third hash value and the new storage node corresponding to the third hash value.
In order to achieve the above object, an embodiment of the present invention further provides a data obtaining apparatus, which is applied to a proxy server in a distributed hash table DHT network, where the proxy server is connected to a plurality of storage nodes, and the apparatus includes:
the first receiving module is used for receiving a first data request sent by a client, wherein the first data request comprises first data information of first data to be requested;
the first determining module is used for determining a first hash value of the first data information according to a preset hash algorithm;
the second determining module is used for determining a first storage node corresponding to the first hash value according to the corresponding relation between the pre-stored hash value and the storage node;
a first sending module, configured to send the first data request to the first storage node, so that the first storage node feeds back an indication message to the proxy server when the first data has been migrated to a second storage node, where the indication message is used to inform the proxy server that the first data has been migrated to the second storage node;
a second sending module, configured to send the first data request to the second storage node according to the indication message;
and the first forwarding module is used for receiving the first data fed back by the second storage node according to the first data request and forwarding the first data to the client.
Optionally, the apparatus further comprises:
a third determining module, configured to determine a third storage node according to a correspondence between preset storage nodes and backup storage nodes, where the number of the backup storage nodes corresponding to the third storage node is smaller than a preset storage number, and the backup storage node corresponding to each storage node is a storage node that backs up original data on the storage node;
a fourth determining module, configured to determine, from the multiple storage nodes connected to the proxy server, a fourth storage node that does not store the original data on the third storage node and has a largest remaining storage space;
and the backup module is used for backing up the original data on the third storage node to the fourth storage node and storing the corresponding relation between the third storage node and the fourth storage node.
Optionally, the apparatus further comprises:
a third sending module, configured to, after sending the first data request to the first storage node, if response information or the indication information for the first data request is not received, send the first data request to the first storage node again, where the response information includes the first data;
a fifth determining module, configured to determine, if no response information or indication information is received in a case of continuously sending a preset number of first data requests, a backup storage node corresponding to a first storage node according to a correspondence between preset stored storage nodes and backup storage nodes, where a backup storage node corresponding to each storage node is a storage node that backs up original data on the storage node;
and the second forwarding module is used for forwarding the first data request to the backup storage node corresponding to the first storage node.
Optionally, the plurality of storage nodes include a physical storage node and a virtual storage node, and the apparatus further includes:
the second receiving module is used for receiving a data storage request, wherein the data storage request comprises second data information of second data to be stored;
a sixth determining module, configured to determine a second hash value of the second data information according to a preset hash algorithm;
a seventh determining module, configured to determine, according to a correspondence between a pre-stored hash value and a storage node, a fifth storage node corresponding to the second hash value;
an eighth determining module, configured to determine, if the fifth storage node is a virtual storage node, a physical storage node corresponding to the fifth storage node according to a correspondence between a pre-stored physical storage node and the virtual storage node;
and the storage module is used for storing the second data into the physical storage node corresponding to the fifth storage node.
Optionally, the apparatus further comprises:
a third receiving module, configured to receive a data migration request sent by a sixth storage node to be exited, where the data migration request includes third data information of each piece of original data stored by the sixth storage node;
a ninth determining module, configured to determine a third hash value of third data information of each piece of original data on the sixth storage node according to a preset hash algorithm;
a tenth determining module, configured to determine, according to a correspondence between pre-stored hash values and storage nodes, a new storage node corresponding to each third hash value as a new storage node corresponding to each original data on the sixth storage node;
a fourth sending module, configured to send a data migration instruction to the sixth storage node, where the data migration instruction includes an identifier of a new storage node corresponding to each piece of original data on the sixth storage node, so that the sixth storage node migrates each piece of original data on the sixth storage node to the corresponding new storage node according to the data migration instruction.
Optionally, the apparatus further comprises:
a fourth receiving module, configured to receive a migration end message sent by the sixth storage node after the data migration instruction is sent to the sixth storage node, where the migration end message indicates that migration of each piece of original data on the sixth storage node is completed;
and the deleting and storing module is used for deleting the corresponding relation between each third hash value and the sixth storage node according to the migration finishing message and storing the corresponding relation between each third hash value and the new storage node corresponding to the third hash value.
In order to achieve the above object, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing any of the above method steps when executing a program stored in the memory.
To achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements any of the above method steps.
An embodiment of the present invention further provides a computer program, which when running on a computer, causes the computer to execute any one of the data acquisition methods described above.
The embodiment of the invention has the following beneficial effects:
according to the data acquisition method, the data acquisition device, the electronic equipment and the storage medium provided by the embodiment of the invention, in the process of migrating data from a first storage node to a second storage node, if a first data request from a proxy server is received, whether the first data stored by the first storage node is migrated to the second storage node is searched. And if the first data is migrated to the second storage node, sending indication information to the proxy server to inform the proxy server that the first data is migrated to the second storage node. And after receiving the indication information, the proxy server sends the first data request to the second storage node, acquires the first data from the second storage node, and then forwards the first data to the client. By the data acquisition method, in the process of migrating data by the first storage node, if the first storage node migrates the first data to the second storage node, the proxy server can acquire the first data from the second storage node, so that the problem that data identification analysis service cannot be provided in the process of migrating data by the first storage node is solved, and the continuity of the data identification analysis service of the nodes in the DHT network is improved.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention;
fig. 2a is an interaction diagram of a DHT network, a proxy server and a client according to an embodiment of the present invention;
fig. 2b is another interaction diagram of a DHT network, a proxy server and a client according to an embodiment of the present invention;
fig. 3 is an interaction diagram of the DHT network, the proxy server and the client when a new node joins according to an embodiment of the present invention;
fig. 4 is a schematic flowchart of a data backup method according to an embodiment of the present invention;
fig. 5 is a schematic partial flowchart of a data acquisition method according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a DHT network according to an embodiment of the present invention;
fig. 7 is a schematic flowchart of a data storage method according to an embodiment of the present invention;
fig. 8 is a flowchart illustrating a data migration method according to an embodiment of the present invention;
fig. 9 is an interaction diagram of a plurality of storage nodes in a DHT network according to an embodiment of the present invention;
fig. 10 is a schematic flowchart of another data migration method according to an embodiment of the present invention;
fig. 11 is a structural diagram of a data acquisition apparatus according to an embodiment of the present invention;
fig. 12 is a structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The DHT network is realized by using a hash algorithm, and comprises a plurality of storage nodes, wherein each storage node has a corresponding hash value. The DHT network may be understood as a ring data structure formed by arranging a plurality of storage nodes in a clockwise direction according to the sizes of hash values corresponding to the storage nodes. When data are stored in the DHT network, the hash value of the data is calculated according to a preset hash algorithm, then the position in the annular data structure corresponding to the hash value of the data is determined, and then the next storage node corresponding to the position corresponding to the data is selected as the storage node of the data along the clockwise direction, so that the storage node of the data can be determined according to the hash value corresponding to the data.
In order to improve the continuity of the data identifier parsing service of the node in the DHT network, an embodiment of the present invention provides a data obtaining method, where the data identifier parsing service may be understood as parsing a data request and providing data corresponding to the data request to a client after the node receives the data request. The data acquisition method provided by the embodiment of the invention is applied to the proxy server in the DHT network, and the proxy server is connected with a plurality of storage nodes. The data acquisition method provided by the embodiment of the invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, a data acquisition method provided in an embodiment of the present invention includes the following steps.
Step 101, receiving a first data request sent by a client, where the first data request includes first data information of first data to be requested.
In the embodiment of the invention, after a user requests first data from a client, the client sends a first data request aiming at the first data to a proxy server, wherein the first data request comprises first data information of the first data, and the first data information is used for determining a hash value corresponding to the first data. The first data includes, but is not limited to, url (Uniform Resource Locator) address.
Step 102, determining a first hash value of the first data information according to a preset hash algorithm.
In the embodiment of the invention, after the proxy server acquires the first data information, the first data information is used as an output parameter of a preset hash algorithm, and the preset hash algorithm is input to obtain a first hash value of the first data information. The predetermined hash algorithm includes, but is not limited to, a hybrid hash algorithm and a bit operation hash algorithm.
Step 103, determining a first storage node corresponding to the first hash value according to the correspondence between the pre-stored hash value and the storage node.
In the embodiment of the present invention, the proxy server stores the correspondence between the hash value and the storage node in advance, and the correspondence between the hash value and the storage node may be that each storage node corresponds to one hash value. At this time, step 103 may be subdivided into that the plurality of storage nodes are arranged in the ring data structure along the clockwise direction according to the size of the corresponding hash value, the position of the first hash value in the ring data structure is determined, the position of the first hash value in the ring data structure is taken as a starting point, the search is performed along the clockwise direction, and the first searched storage node is determined as the first storage node.
The corresponding relationship between the hash value and the storage node may also be that each storage node corresponds to a hash value interval, and the hash value interval corresponding to each storage node is a hash value interval formed by taking the hash values corresponding to all the original data in the storage node as elements. The original data of the storage node is data in the storage node that can provide a data parsing service, that is, data that the storage node can provide to a client based on a data request. In this case, step 103 may be further refined to determine a hash value interval to which the first hash value belongs after determining the first hash value corresponding to the first data, where the storage node corresponding to the hash value interval is the first storage node.
For example, if the hash value interval corresponding to the storage node a is 1-5, the hash value interval corresponding to the storage node B is 6-9, and the hash value interval corresponding to the storage node C is 10-12. If the first hash value is 8, it may be determined that the hash value interval to which 8 belongs is 6-9, and the hash value interval 6-9 corresponds to the storage node B, and then it may be determined that the storage node B is the first storage node.
The length of the hash value interval corresponding to the plurality of storage nodes may be determined based on the location distribution of the plurality of storage nodes in the DHT network, and the lengths of the hash value intervals corresponding to the plurality of storage nodes may be the same or different.
Step 104, sending a first data request to the first storage node, so that the first storage node feeds back an indication message to the proxy server when the first data has been migrated to the second storage node, where the indication message is used to inform the proxy server that the first data has been migrated to the second storage node.
In the embodiment of the present invention, when the first storage node needs to exit the DHT network, the first storage node needs to migrate the original data stored by itself to another storage node.
In the embodiment of the present invention, the second storage node is a storage node that receives data migrated from the first storage node. The second storage node may be a next hop of the first storage node in the DHT network, that is, in the above ring data structure, the first storage node looks up the first storage node clockwise with itself as a starting point. The second storage node may also be determined from the hash value of the original data in the first storage node. Specifically, the proxy server may obtain hash values corresponding to a plurality of pieces of raw data in the first storage node, and then use, as the second storage node of the raw data, a storage node in the DHT network, except the first storage node, that is closest to the hash value corresponding to each piece of raw data in the plurality of pieces of raw data.
In the embodiment of the present invention, the first storage node includes a plurality of original data, and therefore when the first storage node receives the first data request sent by the proxy server, the first data may not have been migrated, and may have been migrated to the second storage node. Therefore, the first storage node detects whether the first data has been migrated to the second storage node after receiving the first data request sent by the proxy server. And if the first data is migrated to the second storage node, sending indication information to the proxy server to inform the proxy server that the first data is migrated to the second storage node, so that the proxy server forwards the first data request to the second storage node.
If the first storage node detects that the first data is not migrated to the second storage node, response information containing the first data can be sent to the proxy server, so that the proxy server extracts the first data from the response information after receiving the response information and forwards the first data to the client.
Step 105, according to the indication message, sending a first data request to the second storage node.
In the embodiment of the invention, after receiving the indication information, the proxy server determines that the first data is migrated to the second storage node, and forwards the first data request to the second storage node.
And 106, receiving first data fed back by the second storage node according to the first data request, and forwarding the first data to the client.
In the embodiment of the invention, after receiving the first data request, the second storage node feeds back response information containing the first data to the proxy server, after receiving the response information, the proxy server extracts the first data from the response information, then forwards the first data to the client, and the client acquires the first data.
According to the data acquisition method provided by the embodiment of the invention, in the process of migrating data, if the first storage node migrates the first data to the second storage node, the first storage node feeds back the indication information to the proxy server, so that the proxy server can acquire the first data from the second storage node.
For example, as shown in fig. 2a, the DHT network includes a storage node 1 and a storage node 2, the storage node 1 stores original data a, b, c, d, and e, and during the process that the storage node 1 migrates the original data to the storage node 2, the proxy server sends a data request for the data d to the storage node 1, at this time, the original data a, b, and c have migrated to the storage node 2, but the original data d is still stored in the storage node 1, the storage node 1 feeds back response information including the data d to the proxy server, and the proxy server sends the data d to the client. On the other hand, as shown in fig. 2b, when the proxy server sends a data request for data a to the storage node 1, since the data a has migrated into the storage node 2, the storage node 1 sends indication information to the proxy server, so that the proxy server forwards the data request for the data a to the storage node 2, and after receiving the data request, the storage node 2 sends response information containing the data a to the proxy server.
In addition, the method can also be applied to a scene that a new node joins the DHT network. As shown in fig. 3, the storage node 2 stores original data a, b, and c, when a new storage node 1 joins the DHT network, in order to reduce the load of the storage node 2, a part of the original data in the storage node 2, such as the original data c, is migrated into the storage node 1, during data migration, when a proxy server sends a data request for the data c to the storage node 2, since the data c has been migrated into the storage node 1, the storage node 2 sends instruction information to the proxy server, so that the proxy server forwards the data request for the data c to the storage node 1, and after receiving the data request for the data c, the storage node 1 sends response information including the data c to the proxy server. The storage node 2 can still provide data identification analysis service during data migration, and the continuity of the data identification analysis service during the process that the storage node migrates data to a new storage node when the new storage node is added to the DHT network is improved.
In an embodiment, as shown in fig. 4, the data acquisition method may further include the following steps.
Step 401, determining a third storage node according to a preset stored correspondence between storage nodes and backup storage nodes, where the number of backup storage nodes corresponding to the third storage node is less than the preset storage number, and the backup storage node corresponding to each storage node is a storage node that backs up original data on the storage node.
In the embodiment of the invention, the proxy server stores the corresponding relation between the storage node and the backup storage node in advance, wherein the backup storage node corresponding to the storage node is the storage node backed up with the original data in the storage node. The proxy server regularly detects the number of backup storage nodes corresponding to each storage node in the corresponding relationship between the storage nodes and the backup storage nodes. When it is detected that the number of backup storage nodes corresponding to a certain storage node is smaller than the preset storage number, it indicates that the original data in the storage node is not backed up or the number of corresponding backup data is small, that is, the proxy server needs to back up the original data in the storage node (e.g., a third storage node). The preset storage amount may be set according to an actual situation, for example, the preset storage amount is 2, 3, or 4, and the like, which is not specifically limited in this embodiment of the present invention.
Step 402, determining a fourth storage node which does not store original data on the third storage node and has the largest residual storage space from a plurality of storage nodes connected by the proxy server.
In an embodiment, after the proxy server determines a third storage node to be backed up from the plurality of storage nodes, in order to facilitate operations and reduce the complexity of data backup, a neighboring node of the third storage node may be directly determined as a fourth storage node, or one storage node may be randomly selected from the DHT network as the fourth storage node, and then original data in the third storage node is backed up in the fourth storage node.
In another embodiment, in order to average the load of each storage node in the DHT network and prevent a certain storage node from having a response delay or even failing to respond due to an excessive load, the proxy server obtains the storage space remaining in each storage node connected to the proxy server after determining a third storage node that needs to be backed up, then determines a storage node with the largest remaining storage space among the plurality of storage nodes as a fourth storage node, and backs up the original data in the third storage node to the fourth storage node.
In this embodiment of the present invention, the fourth storage node may also be determined in other manners, for example, the fourth storage node is determined based on a hash value of the original data in the third storage node and hash values of the original data stored in other storage nodes, which is not specifically limited in this embodiment of the present invention.
Step 403, backing up the original data on the third storage node to the fourth storage node, and storing the corresponding relationship between the third storage node and the fourth storage node.
In the embodiment of the present invention, after the proxy server backs up the original data in the third storage node to the fourth storage node, the original data in the third storage node is the backup data in the fourth storage node, and the fourth storage node is the backup storage node corresponding to the third storage node. And the proxy server records the corresponding relation between the third storage node and the fourth storage node so as to update the corresponding relation between the pre-stored storage node and the backup storage node.
In the embodiment of the invention, the mutual backup of the original data among a plurality of storage nodes is realized by backing up the storage data in the storage nodes to other storage nodes in the DHT network. When a certain storage node in the DHT network fails, any backup storage node corresponding to the failed storage node may be reset to a new storage node corresponding to the original data in the failed storage node. When the proxy server receives the data request aiming at the original data in the failed storage node, the data request can be forwarded to the corresponding new storage node, and the loss of the data in the DHT network is effectively avoided.
In an embodiment, based on the flow shown in fig. 1, after the step 104 sends the first data request to the first storage node, as shown in fig. 5, the data obtaining method provided in the embodiment of the present invention further includes the following steps.
Step 501, if the response information or the indication information for the first data request is not received, the first data request is sent to the first storage node again, and the response information includes the first data.
In the embodiment of the invention, the proxy server sends the first data request to the first storage node, and if the first storage node does not feed back the response message or the indication message to the proxy server, the first storage node may have a fault. In this case, the proxy server resends the first data request to the first storage node in order to further determine whether the first storage node has failed.
Step 502, if no response information or indication information is received when a preset number of first data requests are continuously sent, determining a backup storage node corresponding to a first storage node according to a correspondence between preset stored storage nodes and backup storage nodes, where the backup storage node corresponding to each storage node is a storage node for backing up original data on the storage node.
In the embodiment of the present invention, when the proxy server does not receive the response information or the indication information for the first data request and resends a preset number of first data requests to the first storage node, the proxy server continues to send the first data request to the first storage node if the response information or the indication information fed back by the first storage node is still not received.
If the proxy server does not receive the response information or the indication information fed back by the first storage node after continuously sending the preset number of first data requests to the first storage node, it indicates that the first storage node fails. The proxy server can set the first storage node as unavailable, and then determine one or more backup storage nodes corresponding to the first storage node according to the pre-stored correspondence between the storage nodes and the backup storage nodes. If the backup storage node corresponding to the first storage node stores the first data, it indicates that the backup storage node can feed back response information containing the first data to the proxy server according to the first data request. Therefore, the proxy server sets a backup storage node corresponding to the first storage node as a new storage node of the first data, and is used for responding to the first data request. Wherein the first data represents the original data in the first storage node.
Step 503, forwarding the first data request to a backup storage node corresponding to the first storage node.
In the embodiment of the invention, after the proxy server sets a backup storage node corresponding to the first storage node as a new storage node of the first data, the proxy server sends the first data request to the new storage node of the first data, so that the new storage node of the first data feeds back response information containing the first data to the proxy server. At this time, because the first storage node fails, the proxy server removes the first storage node and the backup storage node corresponding to the first storage node from the corresponding relationship between the storage node and the backup node, so as to update the corresponding relationship between the storage node and the backup node.
For example, as shown in fig. 6, there are 6 storage nodes in the DHT network, and the hash values corresponding to the six storage nodes are 0, 5, 9, 12, 18, and 24, respectively. For convenience of description, the storage node is represented by a hash value corresponding to each storage node. Data in the hash value range of 1-5 is defined as A, data in the hash value range of 6-9 is defined as B, data in the hash value range of 10-12 is defined as C, data in the hash value range of 13-18 is defined as D, data in the hash value range of 19-24 is defined as E, and data in the hash value range of 25-0 is defined as E. In order to prevent a storage node in the DHT network from causing a problem and causing loss of original data in the storage node, the storage nodes are backed up with each other, and the correspondence between the storage nodes and the backup storage nodes is shown in table 1.
TABLE 1
Storage node 5(A) 9(B) 12(C) 18(D) 24(E) 0(F)
Backup storage node 1 12 12 18 24 5 9
Backup storage node 2 24 0 5 0 9 18
As shown in table 1, the original data of the storage node 5 is a, and the storage nodes 12 and 14 are backup storage nodes corresponding to the storage node 5; the original data of the storage node 9 is B, and the storage nodes 12 and 0 are backup storage nodes corresponding to the storage node 9; the original data of the storage node 12 is C, and the storage nodes 18 and 5 are backup storage nodes corresponding to the storage node 12; the original data of the storage node 18 is D, and the storage nodes 24 and 0 are backup storage nodes corresponding to the storage node 18; the original data of the storage node 24 is E, and the storage nodes 5 and 9 are backup storage nodes corresponding to the storage node 24; the original data of the storage node 0 is F, and the storage nodes 9 and 18 are backup storage nodes corresponding to the storage node 0.
At this time, if the storage node 12 fails, it indicates that the storage node 12 cannot continue to respond to the data request for the data C. And the proxy server determines that the backup storage nodes corresponding to the storage node 12 are the storage node 18 and the storage node 5 according to the corresponding relationship between the storage node and the backup storage node, sets the storage node 18 or the storage node 5 as a new storage node of the data C, and responds to a data request aiming at the data C. When the proxy server sets the storage node 18 as a new storage node for the data C, the new correspondence relationship between the storage node and the backup storage node is shown in table 2.
TABLE 2
Storage node 5(A) 9(B) Is free of 18(C and D) 24(E) 0(F)
Backup storage node 1 Is free of Is free of Is free of 24 5 9
Backup storage node 2 24 0 5 0 9 18
As shown in table 2, the original data in the storage node 18 is changed from data D to data C and data D, so that the storage node 18 can respond to the data request for data C, and the loss of data C is avoided.
In an embodiment, in order to make the load of the plurality of storage nodes in the DHT network more even, the plurality of storage nodes may include a physical storage node and a virtual storage node, and the data obtaining method provided in the embodiment of the present invention may further include the following steps, as shown in fig. 7.
Step 701, receiving a data storage request, where the data storage request includes second data information of second data to be stored.
In the embodiment of the present invention, one or more virtual storage nodes may be set for each physical storage node in the DHT network, and the virtual hash value of the virtual storage node corresponding to each physical storage node is calculated according to the physical hash value of the physical storage node. However, there are many possibilities for the virtual hash value corresponding to each physical hash value, and therefore, a plurality of virtual storage nodes corresponding to each physical storage node are distributed at various positions in the ring data structure. As shown in fig. 6, the physical storage node 1 corresponds to three virtual storage nodes 1a, 1b and 1c, and the virtual hash values corresponding to 1a, 1b and 1c are different, so that 1a, 1b and 1c are distributed at a position far from the physical storage node 1 in the ring data structure.
The proxy server receives a data storage request, wherein the data storage request contains second data information of second data to be stored, so that the proxy server can determine a second hash value corresponding to the second data based on the second data information.
Step 702, determining a second hash value of the second data information according to a preset hash algorithm.
In the embodiment of the invention, after the proxy server acquires the second data information, the second data information is used as an input parameter of the preset hash algorithm, and the preset hash algorithm is input to obtain the second hash value of the second data information.
And 703, determining a fifth storage node corresponding to the second hash value according to the correspondence between the pre-stored hash values and the storage nodes.
In the embodiment of the present invention, the correspondence between the hash value and the storage node may be that each storage node corresponds to one hash value. At this time, step 703 may be detailed in that the plurality of virtual storage nodes and the physical storage nodes are arranged in the ring data structure along the clockwise direction according to the size of the corresponding hash value, the position of the second hash value in the ring data structure is determined, the position of the second hash value in the ring data structure is taken as a starting point, the search is performed along the clockwise direction, and the first searched storage node is determined as the fifth storage node.
The fifth storage node may be a physical storage node or a virtual storage node, but the virtual storage node does not have a data storage function, and therefore, when the fifth storage node is a physical storage node, the second data is directly stored in the fifth storage node. When the first storage node is a virtual storage node, the storage node of the second data needs to be further determined based on the fifth storage node.
Step 704, if the fifth storage node is a virtual storage node, determining a physical storage node corresponding to the fifth storage node according to a pre-stored correspondence between the physical storage node and the virtual storage node.
In the embodiment of the present invention, if the fifth storage node is a virtual storage node, it indicates that the second data cannot be directly stored in the fifth storage node. The proxy server pre-stores the corresponding relation between the physical storage node and the virtual storage node, determines the physical storage node corresponding to the fifth storage node according to the corresponding relation between the physical storage node and the virtual storage node, and then stores the second data into the physical storage node corresponding to the fifth storage node.
Step 705, storing the second data into the physical storage node corresponding to the fifth storage node.
In the embodiment of the present invention, each physical storage node corresponds to one or more virtual storage nodes, and since hash values of the virtual storage nodes are different and have a large difference, the virtual node corresponding to each physical storage node can be distributed at each location in the DHT network. When new data is stored, if the hash value of the data to be stored corresponds to the position of the virtual storage node, the data is stored in the physical storage node corresponding to the virtual storage node, so that a plurality of data are stored in each physical storage node, and the hash values corresponding to the plurality of data may belong to different hash value intervals. The load of a plurality of physical storage nodes in the DHT network is more uniform, and the phenomenon that a certain physical storage node is too heavy to cause too slow response or failure in response is avoided.
And if the fifth storage node is a physical storage node, directly storing the second data into the fifth storage node.
In an embodiment, as shown in fig. 8, the data obtaining method provided in the embodiment of the present invention may further include the following steps.
Step 801, receiving a data migration request sent by a sixth storage node to be exited, where the data migration request includes third data information of each original data stored by the sixth storage node.
In the embodiment of the present invention, when the plurality of storage nodes include a plurality of virtual storage nodes and a plurality of physical storage nodes, each physical storage node stores a plurality of pieces of original data, and hash values corresponding to the plurality of pieces of original data may belong to a plurality of different hash value intervals. Therefore, when a certain physical storage node, for example, a sixth storage node, needs to exit the DHT network, the multiple pieces of original data in the sixth storage node may be migrated to different physical storage nodes respectively according to the hash value corresponding to each piece of original data in the sixth storage node.
In the embodiment of the present invention, when the sixth storage node needs to exit the DHT network, the sixth storage node sends a data migration request to the proxy server, and the proxy server extracts the third data information of each original data included in the data migration request after receiving the data migration request.
Step 802, determining a third hash value of third data information of each original data on a sixth storage node according to a preset hash algorithm.
In the embodiment of the present invention, after the proxy server obtains the plurality of third data information, a certain third data information in the plurality of third data information is used as an output parameter of a preset hash algorithm, and the preset hash algorithm is input to obtain a third hash value corresponding to the third data information, so as to obtain a third hash value corresponding to each original data in the sixth storage node.
And 803, determining a new storage node corresponding to each third hash value according to the correspondence between the pre-stored hash values and the storage nodes, and using the new storage node corresponding to each original data on the sixth storage node.
In the embodiment of the present invention, after determining the third hash value corresponding to each piece of raw data on the sixth storage node, the proxy server compares the third hash value corresponding to each piece of raw data with hash values corresponding to other storage nodes in the DHT network, where the other storage nodes are storage nodes in the DHT network except the sixth storage node and the virtual storage node corresponding to the sixth storage node. And then the proxy server determines the storage node closest to the third hash value corresponding to the original data as a new storage node of the original data. The storage node closest to the third hash value may be understood as a storage node having the smallest absolute value of the difference between the hash value corresponding to the storage node (physical storage node and virtual storage node) and the third hash value; it can also be understood as a storage node whose hash value interval corresponding to the storage node includes the third hash value.
In the embodiment of the invention, when the new storage node corresponding to the original data is a physical storage node, the original data is migrated to the new storage node, and when the new storage node corresponding to the original data is a virtual storage node, the data is migrated to the physical storage node corresponding to the new storage node.
Step 804, sending a data migration instruction to the sixth storage node, where the data migration instruction includes an identifier of a new storage node corresponding to each original data on the sixth storage node, so that the sixth storage node migrates each original data on the sixth storage node to the corresponding new storage node according to the data migration instruction.
In the embodiment of the present invention, after determining a new storage node corresponding to each piece of original data in a sixth storage node, a proxy server sends a migration instruction including an identifier of the new storage node corresponding to each piece of original data to the sixth storage node, and after receiving the migration instruction, the sixth storage node may compare the identifier of the new storage node with a plurality of storage nodes in a DHT network, and determine a storage node corresponding to each identifier, thereby determining a new storage node corresponding to each piece of original data, and then migrate each piece of original data to the new storage node corresponding to the piece of original data. Therefore, the situation that the original data in the sixth storage node are directly and completely migrated to the adjacent storage nodes, so that the adjacent storage nodes are overloaded and response is delayed or cannot be responded is avoided.
As shown in FIG. 9, physical storage node 1 corresponds to three virtual storage nodes 1a, 1b, and 1 c; the physical storage node 2 corresponds to two virtual storage nodes 2a and ab; physical storage node 0 corresponds to three virtual storage nodes 0a, 0b, and 0 c. The physical storage node 1 includes data a, b, c, d, and e, and when the physical storage node 1 exits the network, the data c, d, and e are migrated to the storage node 2 and the data a and b are migrated to the storage node 0 according to the hash values of a, b, c, d, and e instead of directly migrating all the data to the adjacent storage node 2. The corresponding position of the hash value corresponding to the data a in the DHT network is the position of the virtual storage node 1a, so that the data a is migrated to the storage node 0a closest to the corresponding hash value, and since 0a is the virtual storage node corresponding to the physical storage node 0, the data a is stored in the physical storage node 0 corresponding to the storage node 0 a. The corresponding position of the hash value corresponding to the data e in the DHT network is the position of the virtual storage node 1c, so the data a is migrated to the storage node 2a closest to the hash value, while the data a is stored in the physical storage node 2 because the virtual storage node 2a corresponds to the physical storage node 2.
The original data in the physical storage node 1 are respectively stored in the physical storage node 2 and the physical storage node 0, so that the loads of the physical storage node 2 and the physical storage node 0 are uniformly distributed, and the situation that the physical storage node 2 or the physical storage node 0 responds too slowly or cannot respond due to sudden load increase of the physical storage node 2 or the physical storage node 0 is effectively avoided.
In an embodiment, as shown in fig. 10, after step 804, the data acquisition method provided in the embodiment of the present invention may further include the following steps.
Step 805, receiving a migration end message sent by the sixth storage node, where the migration end message indicates that migration of each piece of original data on the sixth storage node is completed.
In the embodiment of the present invention, after all the original data in the sixth storage node is migrated, the sixth storage node sends a migration completion message to the proxy server to notify that all the original data stored in the proxy server is migrated.
Step 806, according to the migration end message, deleting the corresponding relationship between each third hash value and the sixth storage node, and storing the corresponding relationship between each third hash value and the new storage node corresponding to the third hash value.
In the embodiment of the present invention, after receiving the migration completion instruction sent by the sixth storage node, the proxy server learns that the sixth storage node can exit the DHT network, and deletes the correspondence between the sixth storage node and the third hash value from the correspondence between the self-storage hash value and the storage node, and records the correspondence between the third hash value and the new storage node, so as to update the correspondence between the self-storage hash value and the storage node. Based on this, the sixth storage node completely exits the DHT network.
In order to improve the continuity of the data identifier resolution service of the node in the DHT network, an embodiment of the present invention further provides a data acquisition apparatus, where the data acquisition apparatus is applied to a proxy server in a DHT network, and as shown in fig. 11, the data acquisition apparatus includes:
a first receiving module 1101, configured to receive a first data request sent by a client, where the first data request includes first data information of first data to be requested;
a first determining module 1102, configured to determine a first hash value of the first data information according to a preset hash algorithm;
a second determining module 1103, configured to determine, according to a correspondence between a pre-stored hash value and a storage node, a first storage node corresponding to the first hash value;
a first sending module 1104, configured to send a first data request to a first storage node, so that the first storage node feeds back an indication message to a proxy server when the first data has been migrated to a second storage node, where the indication message is used to inform the proxy server that the first data has been migrated to the second storage node;
a second sending module 1105, configured to send the first data request to the second storage node according to the indication message;
the first forwarding module 1106 is configured to receive first data fed back by the second storage node according to the first data request, and forward the first data to the client.
In one embodiment, the apparatus may further include:
the third determining module is used for determining a third storage node according to the corresponding relation between the preset storage nodes and the backup storage nodes, the number of the backup storage nodes corresponding to the third storage node is smaller than the preset storage number, and the backup storage node corresponding to each storage node is a storage node for backing up original data on the storage node;
the fourth determining module is used for determining a fourth storage node which does not store original data on the third storage node and has the largest residual storage space from the plurality of storage nodes connected with the proxy server;
and the backup module is used for backing up the original data on the third storage node to the fourth storage node and storing the corresponding relation between the third storage node and the fourth storage node.
In one embodiment, the apparatus may further include:
a third sending module, configured to, after sending the first data request to the first storage node, if response information or indication information for the first data request is not received, send the first data request to the first storage node again, where the response information includes the first data;
a fifth determining module, configured to determine, if no response information or indication information is received in a case of continuously sending a preset number of first data requests, a backup storage node corresponding to the first storage node according to a correspondence between preset stored storage nodes and backup storage nodes, where a backup storage node corresponding to each storage node is a storage node that backs up original data on the storage node;
and the second forwarding module is used for forwarding the first data request to the backup storage node corresponding to the first storage node.
In one embodiment, the plurality of storage nodes include a physical storage node and a virtual storage node, and the apparatus may further include:
the second receiving module is used for receiving a data storage request, wherein the data storage request comprises second data information of second data to be stored;
the sixth determining module is used for determining a second hash value of the second data information according to a preset hash algorithm;
a seventh determining module, configured to determine, according to a correspondence between a pre-stored hash value and a storage node, a fifth storage node corresponding to the second hash value;
an eighth determining module, configured to determine, if the fifth storage node is a virtual storage node, a physical storage node corresponding to the fifth storage node according to a correspondence between a pre-stored physical storage node and the virtual storage node;
and the storage module is used for storing the second data into the physical storage node corresponding to the fifth storage node.
In one embodiment, the apparatus may further include:
a third receiving module, configured to receive a data migration request sent by a sixth storage node to be exited, where the data migration request includes third data information of each piece of original data stored by the sixth storage node;
a ninth determining module, configured to determine a third hash value of third data information of each piece of original data on a sixth storage node according to a preset hash algorithm;
a tenth determining module, configured to determine, according to a correspondence between pre-stored hash values and storage nodes, a new storage node corresponding to each third hash value as a new storage node corresponding to each original data on a sixth storage node;
and the fourth sending module is used for sending a data migration instruction to the sixth storage node, wherein the data migration instruction includes an identifier of a new storage node corresponding to each original data on the sixth storage node, so that the sixth storage node migrates each original data on the sixth storage node to the corresponding new storage node according to the data migration instruction.
In one embodiment, the apparatus may further include:
the fourth receiving module is configured to receive a migration end message sent by the sixth storage node after sending the data migration instruction to the sixth storage node, where the migration end message indicates that migration of each piece of original data on the sixth storage node is completed;
and the deleting and storing module is used for deleting the corresponding relation between each third hash value and the sixth storage node according to the migration finishing message and storing the corresponding relation between each third hash value and the new storage node corresponding to the third hash value.
An embodiment of the present invention further provides an electronic device, as shown in fig. 12, including a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204, where the processor 1201, the communication interface 1202, and the memory 1203 complete mutual communication through the communication bus 1204,
a memory 1203 for storing a computer program;
a processor 1201 for implementing the steps of any of the above-described data acquisition methods when executing the program stored in the memory 1203
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program realizes the steps of any one of the above data acquisition methods when executed by a processor.
In a further embodiment of the present invention, there is also provided a computer program which, when run on a computer, causes the computer to perform any one of the data acquisition methods of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program. The computer program includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus, the electronic device, the computer-readable storage medium, and the computer program, since they are substantially similar to the method embodiments, the description is relatively simple, and in relation to the description, reference may be made to some portions of the description of the method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims (10)

1. A data acquisition method is applied to a proxy server in a DHT network, wherein the proxy server is connected with a plurality of storage nodes, and the method comprises the following steps:
receiving a first data request sent by a client, wherein the first data request comprises first data information of first data to be requested;
determining a first hash value of the first data information according to a preset hash algorithm;
determining a first storage node corresponding to the first hash value according to a corresponding relation between the pre-stored hash value and the storage node;
sending the first data request to the first storage node, so that the first storage node feeds back an indication message to the proxy server when the first data is migrated to a second storage node, wherein the indication message is used for informing the proxy server that the first data is migrated to the second storage node;
sending the first data request to the second storage node according to the indication message;
and receiving the first data fed back by the second storage node according to the first data request, and forwarding the first data to the client.
2. The method of claim 1, further comprising:
determining a third storage node according to a corresponding relation between preset storage nodes and backup storage nodes, wherein the number of the backup storage nodes corresponding to the third storage node is smaller than the preset storage number, and the backup storage node corresponding to each storage node is a storage node for backing up original data on the storage node;
determining a fourth storage node which does not store original data on the third storage node and has the largest residual storage space from a plurality of storage nodes connected with the proxy server;
and backing up the original data on the third storage node to the fourth storage node, and storing the corresponding relation between the third storage node and the fourth storage node.
3. The method of claim 1, wherein after sending the first data request to the first storage node, the method further comprises:
if response information or the indication information aiming at the first data request is not received, the first data request is sent to the first storage node again, and the response information comprises the first data;
if a preset number of first data requests are continuously sent and no response information or indication information is received, determining backup storage nodes corresponding to the first storage nodes according to the corresponding relation between preset storage nodes and backup storage nodes, wherein the backup storage node corresponding to each storage node is a storage node for backing up original data on the storage node;
and forwarding the first data request to a backup storage node corresponding to the first storage node.
4. The method of claim 1, wherein the plurality of storage nodes comprises physical storage nodes and virtual storage nodes, the method further comprising:
receiving a data storage request, wherein the data storage request comprises second data information of second data to be stored;
determining a second hash value of the second data information according to a preset hash algorithm;
determining a fifth storage node corresponding to the second hash value according to the corresponding relation between the pre-stored hash value and the storage node;
if the fifth storage node is a virtual storage node, determining a physical storage node corresponding to the fifth storage node according to a pre-stored corresponding relationship between the physical storage node and the virtual storage node;
and storing the second data into a physical storage node corresponding to the fifth storage node.
5. The method of claim 1, further comprising:
receiving a data migration request sent by a sixth storage node to be exited, wherein the data migration request comprises third data information of each original data stored by the sixth storage node;
determining a third hash value of third data information of each original data on the sixth storage node according to a preset hash algorithm;
determining a new storage node corresponding to each third hash value according to a correspondence between a pre-stored hash value and a storage node, and using the new storage node as a new storage node corresponding to each original data on the sixth storage node;
sending a data migration instruction to the sixth storage node, where the data migration instruction includes an identifier of a new storage node corresponding to each piece of original data on the sixth storage node, so that the sixth storage node migrates each piece of original data on the sixth storage node to the corresponding new storage node according to the data migration instruction.
6. The method of claim 5, wherein after said sending a data migration instruction to the sixth storage node, the method further comprises:
receiving a migration end message sent by the sixth storage node, where the migration end message indicates that migration of each original data on the sixth storage node is completed;
and deleting the corresponding relation between each third hash value and the sixth storage node according to the migration ending message, and storing the corresponding relation between each third hash value and the new storage node corresponding to the third hash value.
7. A data acquisition apparatus, applied to a proxy server in a distributed hash table DHT network, the proxy server being connected to a plurality of storage nodes, the apparatus comprising:
the system comprises a receiving module, a sending module and a receiving module, wherein the receiving module is used for receiving a first data request sent by a client, and the first data request comprises first data information of first data to be requested;
the first determining module is used for determining a first hash value of the first data information according to a preset hash algorithm;
the second determining module is used for determining a first storage node corresponding to the first hash value according to the corresponding relation between the pre-stored hash value and the storage node;
a first sending module, configured to send the first data request to the first storage node, so that the first storage node feeds back an indication message to the proxy server when the first data has been migrated to a second storage node, where the indication message is used to inform the proxy server that the first data has been migrated to the second storage node;
a second sending module, configured to send the first data request to the second storage node according to the indication message;
and the forwarding module is used for receiving the first data fed back by the second storage node according to the first data request and forwarding the first data to the client.
8. The apparatus of claim 7, further comprising:
a third determining module, configured to determine a third storage node according to a correspondence between preset storage nodes and backup storage nodes, where the number of the backup storage nodes corresponding to the third storage node is smaller than a preset storage number, and the backup storage node corresponding to each storage node is a storage node that backs up original data on the storage node;
a fourth determining module, configured to determine, from the multiple storage nodes connected to the proxy server, a fourth storage node that does not store the original data on the third storage node and has a largest remaining storage space;
and the backup module is used for backing up the original data on the third storage node to the fourth storage node and storing the corresponding relation between the third storage node and the fourth storage node.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 6 when executing a program stored in the memory.
10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1-6.
CN202011403471.9A 2020-12-02 2020-12-02 Data acquisition method and device, electronic equipment and storage medium Pending CN112511634A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011403471.9A CN112511634A (en) 2020-12-02 2020-12-02 Data acquisition method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011403471.9A CN112511634A (en) 2020-12-02 2020-12-02 Data acquisition method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN112511634A true CN112511634A (en) 2021-03-16

Family

ID=74968412

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011403471.9A Pending CN112511634A (en) 2020-12-02 2020-12-02 Data acquisition method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112511634A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112988471A (en) * 2021-04-27 2021-06-18 北京邮电大学 Data backup method and device in industrial Internet system and electronic equipment
CN113486051A (en) * 2021-06-16 2021-10-08 中国联合网络通信集团有限公司 Industrial internet identification analysis method, system, node and storage medium
CN114500576A (en) * 2021-12-27 2022-05-13 天翼云科技有限公司 Distributed cache capacity expansion and reduction method, system, device and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103516755A (en) * 2012-06-27 2014-01-15 华为技术有限公司 Virtual storage method and equipment thereof
US20170270007A1 (en) * 2015-12-03 2017-09-21 Huawei Technologies Co.,Ltd. Method, apparatus, and system for migrating virtual machine backup information
CN109189324A (en) * 2018-07-09 2019-01-11 华为技术有限公司 A kind of data migration method and device
CN110049091A (en) * 2019-01-10 2019-07-23 阿里巴巴集团控股有限公司 Date storage method and device, electronic equipment, storage medium
CN111629051A (en) * 2020-05-26 2020-09-04 北京邮电大学 Performance optimization method and device for industrial internet identification analysis system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103516755A (en) * 2012-06-27 2014-01-15 华为技术有限公司 Virtual storage method and equipment thereof
US20170270007A1 (en) * 2015-12-03 2017-09-21 Huawei Technologies Co.,Ltd. Method, apparatus, and system for migrating virtual machine backup information
CN109189324A (en) * 2018-07-09 2019-01-11 华为技术有限公司 A kind of data migration method and device
CN110049091A (en) * 2019-01-10 2019-07-23 阿里巴巴集团控股有限公司 Date storage method and device, electronic equipment, storage medium
CN111629051A (en) * 2020-05-26 2020-09-04 北京邮电大学 Performance optimization method and device for industrial internet identification analysis system

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112988471A (en) * 2021-04-27 2021-06-18 北京邮电大学 Data backup method and device in industrial Internet system and electronic equipment
CN112988471B (en) * 2021-04-27 2023-03-14 北京邮电大学 Data backup method and device in industrial Internet system and electronic equipment
CN113486051A (en) * 2021-06-16 2021-10-08 中国联合网络通信集团有限公司 Industrial internet identification analysis method, system, node and storage medium
CN113486051B (en) * 2021-06-16 2023-06-13 中国联合网络通信集团有限公司 Industrial Internet identification analysis method, system, node and storage medium
CN114500576A (en) * 2021-12-27 2022-05-13 天翼云科技有限公司 Distributed cache capacity expansion and reduction method, system, device and storage medium

Similar Documents

Publication Publication Date Title
CN112511634A (en) Data acquisition method and device, electronic equipment and storage medium
CN108810100B (en) Method, device and equipment for electing master node
CN110737668B (en) Data storage method, data reading method, related device and medium
CN108696581B (en) Distributed information caching method and device, computer equipment and storage medium
EP2948875B1 (en) Method and system for using a recursive event listener on a node in hierarchical data structure
CN106712981B (en) Node change notification method and device
CN106230997B (en) Resource scheduling method and device
CN109921925B (en) Dial testing method and device
CN103458013A (en) Streaming media server cluster load balancing system and balancing method
CN103546572A (en) Cloud storage device and multi-cloud storage networking system and method
JP2016162389A (en) Thin client system, connection management device, virtual machine operating device, method, and program
CN108509296B (en) Method and system for processing equipment fault
US10853892B2 (en) Social networking relationships processing method, system, and storage medium
US10904327B2 (en) Method, electronic device and computer program product for searching for node
KR102631288B1 (en) Subscription server, subscription terminal, information subscription method, and system
CN101605094B (en) Ring model based on point-to-point network and routing algorithm thereof
US20090132716A1 (en) Fault-tolerant distributed services methods and systems
CN114338794B (en) Service message pushing method and device, electronic equipment and storage medium
CN111262916B (en) Data distributed storage method and device, storage node and storage medium
CN114048228A (en) State storage updating method, device, equipment and storage medium
CN108234173B (en) Node copy updating method and device
JP2016157367A (en) Distributed processing system
CN110221916B (en) Memory capacity expansion method and device, configuration center system and electronic equipment
CN106156166B (en) Relation chain query system, document retrieval method, index establishment method and device
CN112988471B (en) Data backup method and device in industrial Internet system and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20210316