CN117348806A - Digital object migration method, device and storage medium - Google Patents

Digital object migration method, device and storage medium Download PDF

Info

Publication number
CN117348806A
CN117348806A CN202311264989.2A CN202311264989A CN117348806A CN 117348806 A CN117348806 A CN 117348806A CN 202311264989 A CN202311264989 A CN 202311264989A CN 117348806 A CN117348806 A CN 117348806A
Authority
CN
China
Prior art keywords
storage
storage node
block
node
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311264989.2A
Other languages
Chinese (zh)
Inventor
朱晓旻
戎劲光
但林
郭京申
高晶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanxi Yunshidai Technology Co ltd
Beijing Zhirong Yunhe Technology Co ltd
Original Assignee
Shanxi Yunshidai Technology Co ltd
Beijing Zhirong Yunhe Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanxi Yunshidai Technology Co ltd, Beijing Zhirong Yunhe Technology Co ltd filed Critical Shanxi Yunshidai Technology Co ltd
Priority to CN202311264989.2A priority Critical patent/CN117348806A/en
Publication of CN117348806A publication Critical patent/CN117348806A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3034—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a storage system, e.g. DASD based or network based
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629—Configuration or reconfiguration of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/542—Event management; Broadcasting; Multicasting; Notifications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Computer Security & Cryptography (AREA)
  • Multimedia (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a digital object migration method, a digital object migration device and a storage medium, and belongs to the technical field of data processing. Monitoring the change of a cluster topology of a storage cluster, wherein the storage cluster comprises a plurality of storage nodes, and the cluster topology comprises all online nodes of the storage cluster at present; when the change of the cluster topology of the storage cluster is monitored to be any storage node to exit, respectively determining the corresponding target storage nodes for all the blocks stored in the storage node; when monitoring that the change of the cluster topology of the storage cluster is to join a new storage node, the new storage node determines whether the new storage node is a target storage node of any block or not; and the target storage node executes a data migration strategy, and acquires a block corresponding to the target storage node for storage, wherein any block is migrated at most once when the data migration strategy is executed. The application aims to improve the security of data backup.

Description

Digital object migration method, device and storage medium
Technical Field
The embodiment of the application relates to the technical field of data processing, in particular to a digital object migration method, a digital object migration device and a storage medium.
Background
With the rapid development of digitization, more and more services have been abstracted into data services, such as the widespread use of service systems of digital networking, and it is important to face a large number of data elements circulated in various data services, and how to effectively store the data elements.
In the field of data storage, in order to improve the security of data storage, a backup storage mode is generally adopted, for example, in a storage cluster with multiple storage nodes, a plurality of storage nodes can be backed up for one piece of data at the same time, for example, a blockchain is used as a mode of data circulation, and has the characteristics of data falsification and decentralization.
However, in the existing process of backing up data by adopting a plurality of storage nodes, the influence on the data backup caused by the dynamic change of the storage nodes in the storage cluster is ignored, the number of the data backups caused by the dynamic change of the storage nodes cannot meet the requirement, even the data backups are lost, and the safety of the data backups is reduced.
Disclosure of Invention
The embodiment of the application provides a digital object migration method, a digital object migration device and a storage medium, aiming at improving the security of data backup.
In a first aspect, an embodiment of the present application provides a digital object migration method applied to a storage cluster of a distributed ledger, where the method includes:
monitoring the change of a cluster topology of the storage cluster, wherein the storage cluster comprises a plurality of storage nodes, and the cluster topology comprises all online nodes of the storage cluster at present;
when the change of the cluster topology of the storage cluster is monitored to be any storage node to exit, respectively determining the corresponding target storage nodes for all the blocks stored in the storage node;
when monitoring that the change of the cluster topology of the storage cluster is to join a new storage node, the new storage node determines whether the new storage node is a target storage node of any block or not;
and the target storage node executes a data migration strategy, and acquires a block corresponding to the target storage node for storage, wherein any block is migrated at most once when the data migration strategy is executed.
Optionally, monitoring the storage cluster for a change in cluster topology, including:
Monitoring an exit request of any node in a cluster topology of the storage cluster; or alternatively, the first and second heat exchangers may be,
any storage node in the storage cluster sends heartbeat information to a system chain of the distributed account book according to a preset time interval, and the survival identification of the storage node on the system chain is increased by 1;
each time a block is produced, determining the respective survival identifications of all storage nodes on the system chain, and subtracting 1 from the survival identifications of all storage nodes;
and when the survival identification of any storage node on the system chain is 0 during each block generation, rejecting the storage node in the cluster topology of the storage cluster.
Optionally, the method further comprises:
updating the latest version number of the cluster topology when the cluster topology of the storage cluster changes;
when any storage node in the storage cluster exits, the version number of the cluster topology at the time of exiting is locally stored in the storage node;
when the storage node joins the storage cluster again, comparing the version number of the cluster topology stored locally in the storage node with the latest version number of the cluster topology;
when the version number of the locally stored cluster topology is the same as the latest version number of the cluster topology, the storage node starts from the locally stored block scanning offset to determine whether the storage node is a target storage node of any block or not;
When the version number of the locally stored cluster topology is smaller than the latest version number of the cluster topology, the storage node starts scanning all the blocks from 0 and determines whether the storage node is a target storage node of any block or not.
Optionally, the method further comprises:
dividing a plurality of storage nodes of the storage cluster into a plurality of storage layers;
when monitoring that the change of the cluster topology of the storage cluster is that any storage node exits, determining the corresponding target storage node for all the blocks stored in the storage node respectively comprises the following steps:
when any storage node exits, for each storage node in the storage layer to which the storage node belongs, determining a block hash of each block stored in the storage node and hash values of node identifiers of all storage nodes in the storage layer, and determining whether the block hash value is a target storage node of the block according to the ordering of the hash values.
Optionally, the method further comprises:
dividing a plurality of storage nodes of the storage cluster into a plurality of storage layers;
when monitoring that the change of the cluster topology of the storage cluster is to join a new storage node, the new storage node determines whether the new storage node is a target storage node of any block or not, including:
The new storage node scans all blocks from 0 to acquire each target block stored in a storage layer to which the new storage node belongs;
and the new storage node determines the block hash of each target block and the hash value of the node identification of all the storage nodes in the storage layer to which the new storage node belongs, and determines whether the new storage node is the target storage node of the target block according to the ordering of the hash values.
Optionally, the target storage node executes a data migration policy, including:
the target storage node determines a candidate node list of a block to be migrated, wherein the candidate node list of the block to be migrated comprises a plurality of storage nodes for storing the block to be migrated;
the target storage node selects one storage node in the candidate node list to send a block request message of a block to be migrated;
and if the response of the block request message is not received, the target storage node selects the next storage node in the candidate node list to send the block request message of the block to be migrated.
Optionally, the target storage node executes a data migration policy, and further includes:
and the target storage node broadcasts and searches all the storage nodes in the candidate node list for the block to be migrated until the block request message of the block to be migrated is sequentially sent to all the storage nodes in the candidate node list and the response of the block request message is not received.
Optionally, the target storage node selects one storage node in the candidate node list to send a block request message of a block to be migrated, and further includes:
the target storage node selects one storage node in the candidate node list and establishes communication flow with the storage node;
and maintaining the communication flow established between the target storage node and the storage node, taking the storage node as a neighbor node of the target storage node, and when a candidate node list of any block to be migrated of the target storage block contains the neighbor node, sending a block request message of the block to be migrated to the neighbor node by the target storage block with the highest priority.
In a second aspect, an embodiment of the present application provides a digital object migration apparatus, applied to a storage cluster of a distributed ledger, where the apparatus includes:
the monitoring module is used for monitoring the change of the cluster topology of the storage cluster, wherein the storage cluster comprises a plurality of storage nodes, and the cluster topology comprises all online nodes of the storage cluster at present;
the first target storage node determining module is used for determining the corresponding target storage nodes for all the blocks stored in the storage nodes respectively when monitoring that the change of the cluster topology of the storage cluster is that any storage node exits;
The second target storage node determining module is used for determining whether the new storage node is a target storage node of any block or not when the change of the cluster topology of the storage cluster is monitored to be a new storage node;
and the migration module is used for executing the data migration strategy by the target storage node, acquiring the block corresponding to the target storage node and storing the block, wherein any block is migrated at most once when the data migration strategy is executed.
In a third aspect, embodiments of the present application provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the digital object migration method according to the first aspect.
The beneficial effects are that:
monitoring the change of a cluster topology of a storage cluster, wherein the cluster topology comprises all current online nodes of the storage cluster; when the change of the cluster topology of the storage cluster is monitored to be any storage node to exit, respectively determining the corresponding target storage nodes for all the blocks stored in the storage node; when monitoring that the change of the cluster topology of the storage cluster is to join a new storage node, the new storage node determines whether the new storage node is a target storage node of any block or not; the target storage node executes the data migration strategy, acquires the block corresponding to the target storage node for storage, and any block is migrated at most once when the data migration strategy is executed.
When a storage node exits, the storage node is reselected in time for backup of the block stored in the exiting storage node, so that the situation that the backup of the data of the block does not meet the requirement and even the data of the block is lost is avoided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart illustrating steps of a method for migrating a digital object according to one embodiment of the present application;
FIG. 2 is a schematic diagram of a storage node selection result according to an embodiment of the present application;
fig. 3 is a functional block diagram of a digital object migration apparatus according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
Referring to fig. 1, a flowchart illustrating steps of a digital object migration method in an embodiment of the present application, where the method is applied to a storage cluster of a distributed ledger, the method may specifically include the steps of:
s101: monitoring the change of a cluster topology of the storage cluster, wherein the storage cluster comprises a plurality of storage nodes, and the cluster topology comprises all online nodes of the storage cluster currently.
The storage cluster of the distributed ledger includes a plurality of storage nodes, according to the actual configuration of the distributed ledger by adopting the plurality of storage nodes, the cluster topology of the storage cluster can be obtained, for example, in the actual implementation process, the storage nodes can be classified according to different standards, for example, according to different hardware configurations of the storage nodes, the storage nodes can be divided into a plurality of storage layers, different storage layers are used for storing different blocks, and corresponding storage layers can be determined according to the block production time of the blocks or the importance degree of the data of the blocks, and the embodiment is not limited.
In the distributed ledger technology based on directed acyclic graphs (Directed Acyclic Graph, DAG), for example, the BDLedger ledger also uses the structure of the DAG to organize blocks, so that the transaction processing speed is greatly improved, besides the ledger of the illustrated structure, a system chain is maintained in the BDLedger ledger to manage metadata, and the blocks packed by the system chain according to the sequence are linked together to form a continuously-growing chain structure, so that the system chain based on the BDLedger distributed ledger can maintain and manage the cluster topology of a storage cluster of the distributed ledger, and all nodes can carry out final consistency consensus on the cluster topology through the system chain in the global knowledge consensus stage.
Monitoring the change of the cluster topology of the storage cluster, including monitoring the exit request or the joining request of any node in the cluster topology of the storage cluster; however, in practical application, some storage nodes are not actively logged off and may be offline due to failure, power failure, etc., so that it is also necessary to monitor the change of cluster topology caused by the storage nodes that are not actively logged off.
In a possible implementation manner, for any storage node in the storage cluster, heartbeat information is sent to the system chain of the distributed ledger according to a preset time interval to inform the system chain of the online state of the storage node, and a survival identifier may be set for the online state of one storage node, for example, survival time TTL (Time To Live) is used as the survival identifier.
Specifically, any storage node sends heartbeat information to a system chain according to a preset time interval, and then adds 1 to the survival identifier TTL of the storage node on the system chain; then, each time a block is produced, determining the respective survival identifications of all storage nodes on the system chain, and subtracting 1 from the survival identifications of all storage nodes; when the survival mark of any storage node on the system chain is 0 during each block generation, the storage node is indicated to have no updated on-line state for a long time, and faults possibly exist, so that the storage node is removed from the cluster topology of the storage cluster.
When a new storage node is added, the new storage node sends configuration information of the new storage node to the system chain, the system chain can add the new storage node into the cluster topology, and for a storage cluster divided into a plurality of storage layers, the system chain can divide the new storage node into corresponding storage layers according to the configuration information of the new storage node.
For example, for a storage cluster of a distributed ledger divided into multiple storage layers, the system chain maps on-line storage nodes of multiple different storage layers, such as Tier1: [ node1: TTL, node2: TTL … ] … …, wherein Tier represents a storage layer and node represents a storage node, thereby ensuring that online storage nodes in each storage hierarchy are reflected in the cluster topology.
S102: and when monitoring that the change of the cluster topology of the storage cluster is that any storage node exits, respectively determining the corresponding target storage nodes for all the blocks stored in the storage node.
In a possible implementation manner, taking an example that a plurality of storage nodes of the storage cluster are divided into a plurality of storage layers, when it is monitored that a cluster topology of the storage cluster changes to any storage node to exit, the target storage node needs to be redetermined for each block stored in the exiting storage node.
For decentralization, it may be determined by each storage node in the storage tier to which the retired storage node belongs whether the blocks stored in the retired storage node need to be stored.
Specifically, each storage node in the storage layer to which the storage node belongs acquires the block hash of each block in the withdrawn storage node, for any block, the respective node identification of all the storage nodes in the storage layer to which the storage node belongs can be determined according to the cluster topology acquired on the system chain, then the block hash of the block and the hash value of the respective node identification of all the storage nodes in the storage layer to which the storage node belongs are calculated, the calculated hash values are ordered, and if one block needs to be backed up for storing N pieces, if one storage node judges that the own node identification and the hash value of the block hash of the block are arranged in the first N bits in all the storage nodes in the storage layer, the storage node is taken as the target storage node of the block.
S103: when the change of the cluster topology of the storage cluster is monitored to be a new storage node, the new storage node determines whether the new storage node is a target storage node of any block or not.
In a possible implementation, taking an example that a plurality of storage nodes of the storage cluster are divided into a plurality of storage layers, when it is monitored that a change of a cluster topology of the storage cluster is to join a new storage node, the new storage node scans all blocks from 0, and acquires each target block stored in the storage layer to which the new storage node belongs.
In the practical implementation process, according to different division conditions of a plurality of storage layers of the storage cluster, a mapping relationship between a block and a storage layer can be defined in a block header of the block, for example, when the storage cluster is divided according to the importance degree of data of the block, the importance degree of the block is contained in the block header of the block, and the storage layer for storing the block can be determined by scanning the block header of the block.
And the new storage node determines the hash value of the block hash of each target block and the hash values of the node identifiers of all storage nodes in the storage layer to which the new storage node belongs, determines whether the new storage node is the target storage node of the target block according to the sequence of the hash values, and presumes that N blocks need to be backed up and stored.
In a possible implementation, a version mechanism may also be introduced for the cluster topology of the storage cluster, i.e. when the cluster topology of the storage cluster changes, the latest version number of the cluster topology is updated, for example, the version number of the cluster topology is marked with an integer variable, and the version of the changed cluster topology is recorded with an integer variable increment.
Then when any storage node in the storage cluster exits, the version number of the cluster topology at the time of exiting is locally stored in the storage node; when the storage node joins the storage cluster again, comparing the version number of the cluster topology stored locally in the storage node with the latest version number of the cluster topology.
When the version number of the locally stored cluster topology is the same as the latest version number of the cluster topology, the storage node starts from the locally stored block scanning offset to determine whether the storage node is a target storage node of any block or not; because the version number of the local storage cluster topology is the same as the latest version number of the cluster topology, no node is withdrawn or added in the storage layer, and for the blocks stored in the storage layer before the node is withdrawn, the storage nodes for block backup do not need to be changed, and only whether the newly added blocks after the node is withdrawn are used as the target storage nodes of the newly added blocks or not by the node is needed, so that the blocks are scanned from 0 differently, and a large number of scanning block time is reduced.
When the version number of the locally stored cluster topology is smaller than the latest version number of the cluster topology, the existence of node exit or node joining is indicated, so that the storage node scans all blocks from 0 and determines whether the storage node is a target storage node of any block.
When the version of the cluster topology changes, all storage nodes can be synchronized to the latest version of the cluster topology through all nodes of the system chain; for example, the storage node a exits the cluster topology when the storage node a exits the cluster topology in version 3, and other storage nodes join or exit during the leaving period of the storage node a, if the storage node 3 joins the cluster again, after the storage node is started, the storage nodes will pass through versions 4, 5, 6 and 7 of the cluster topology in sequence in the synchronization state until the latest version of the cluster topology is caught up.
However, if the storage node synchronizes the cluster topology and determines the block storage adjustment under each version, a new version of the cluster topology is continuously generated, and a part of nonsensical overhead is caused by repeatedly generating such a process, so that the historical version of the cluster topology can be ignored, i.e. the cluster version change is started to be processed after the system chain state is timely followed; in the implementation, when the storage node is added, a heartbeat message serving as a Barrier (Barrier) can be sent to the system chain, and the heartbeat message serving as the Barrier is packed into a block of the system chain and is executed locally, so that the local meta-information of the storage node is ensured to reach or be close to a global synchronous state, and then the block storage adjustment under the latest version of the cluster topology is started to be determined.
S104: and the target storage node executes a data migration strategy, and acquires a block corresponding to the target storage node for storage.
In this embodiment, when the cluster topology changes and the target storage node of any block is determined based on the hash value after sorting, one block only needs to be migrated once, so that the data transmission amount can be effectively reduced.
For example, if a storage node exits, block a stored in that node is migrated to only one target storage node, and when a new storage node joins, the new storage node will only replace an old storage node to store the block.
Referring to fig. 2, a schematic diagram of a storage node selection result provided in an embodiment of the present application is shown, as shown in fig. 2 (a), assuming that 3 backup nodes are determined for each block, after a target storage layer corresponding to the block is determined, there are 6 storage nodes in the target storage layer, for a block, after a hash value straw is obtained by performing hash calculation on a node identifier of each node and a block hash of the block, a node with a straw value arranged in the first 3 is selected as an initial storage node, and assuming that the initial storage node includes: node 1, node 4, and node 6.
As shown in fig. 2 (b), the storage node 3 exits, but for the block, the storage node 3 is not originally the initial storage node, so the storage node 3 is not responsible for storing the block, and therefore has no influence on the storage node selection of the block, and the storage result is kept unchanged; the storage of this block is not adjusted, assuming that the subsequent process node 3 rejoins.
As shown in fig. 2 (c), after the storage node 3 exits, the storage node 6 also exits, and the storage node 6 is responsible for backing up the block before, so that after the storage node 6 exits, 1 new target storage node needs to be reselected for the block, and the node with the largest straw value in the remaining nodes in the target storage layer, namely, the storage node 2, is selected as the target storage node by calculating the node identifier of each storage node in the target storage layer and the block hash of the block; and when one storage node of the block exits, only one new node is needed to store the block in a backup mode, so that the data migration quantity is smaller.
As shown in fig. 2 (d), after the storage nodes 3 and 6 exit, the storage node 7 is newly added, the target storage node of the block is newly determined, and the new target storage nodes of the block are determined to be the node 1, the node 4 and the node 7 according to the calculated straw value, and since the block is already stored in the node 1 and the node 4, any data migration is not required, and only the block is required to be migrated to the node 7 for storage.
Therefore, in this embodiment, when the cluster topology changes, whether a storage node exits, a storage node restarts or a new storage node joins, data migration will occur at most once, and as the number of storage nodes increases, the probability of change decreases, so that the migration volume of the block can be minimized.
In a possible implementation manner, when the target storage node executes the data migration policy, the method includes: the target storage node determines a candidate node list of a block to be migrated, wherein the candidate node list of the block to be migrated comprises a plurality of storage nodes for storing the block to be migrated, namely all the storage nodes for storing the block to be migrated are constructed as a candidate node list; then the target storage node selects one storage node in the candidate node list to send a block request message of a block to be migrated; and if the response of the block request message is not received, the target storage node selects the next storage node in the candidate node list to send the block request message of the block to be migrated.
When the block data is migrated, the positions of the storage nodes which are already stored with the blocks are efficiently found, the interaction time with other storage nodes is reduced as much as possible, and the problems that the storage nodes are likely to be in fault, offline, network abnormality and the like are solved, so that when a block to be migrated is obtained, the block can be migrated in a traversal mode by constructing a candidate node list and sending block request information of the block to be migrated to the storage nodes in the candidate node list one by one, and any storage node in the candidate list is likely to exist briefly
And the target storage node broadcasts and searches all the storage nodes in the candidate node list for the block to be migrated until the block request message of the block to be migrated is sequentially sent to all the storage nodes in the candidate node list and the response of the block request message is not received.
In the practical implementation process, there may be a case that a plurality of blocks are acquired simultaneously for one storage node to perform migration backup, in order to reduce the time of data transmission, the storage node may initiate a block request message for a plurality of blocks in parallel, for example, for one storage node in the candidate node list of the block 1 to be migrated, the node does not need to wait for receiving a response of the block request message, and continues to send the block request message to one storage node in the candidate node list of the block 2 to be migrated, and only needs to respond to any one block request message and acquire a block in real time.
In a possible implementation manner, when the target storage node selects one storage node in the candidate node list to send a block request message of a block to be migrated, the method further includes:
the target storage node selects one storage node in the candidate node list and establishes a communication flow with the storage node.
Specifically, when the target storage node initiates a block request message to a storage node in a candidate node list, network communication is to be initiated to the storage node through the id of the storage node, for example, a dedicated libp2p protocol fetchblock is used, which is a protocol for performing block synchronization between nodes in a blockchain network.
In the blockchain, each block contains transaction information and a hash value pointing to the previous block, so to completely synchronize the blockchain, each block on the network needs to be acquired and verified by the protocol FetchBlock protocol.
Specifically, when implementing the protocol FetchBlock protocol, a storage node may send a request to other nodes to request the other nodes to transmit data of a certain block, if the other nodes receive the request, the other nodes transmit the data of the block and related verification information to the requesting party, and after receiving the data, the requesting party needs to verify the block to ensure that the hash value and other information of the block are correct, and then the hash value and other information can be added into own blockchain.
Before establishing a data transmission stream, negotiating a multiplexing protocol through encryption connection, establishing a communication stream based on the protocol to perform data transmission among nodes, when establishing a p2p data stream, firstly selecting a storage node from a candidate node list, then performing storage node routing through the storage node id, inquiring a storage node address to a DHT network to obtain an IP address and a port number of the storage node, negotiating to establish the encryption connection by using a TLS protocol, negotiating the multiplexing protocol of the negotiation stream on the encryption connection, establishing a libp2p data stream, and then initiating a block request message through the data stream according to a block hash, wherein the block request message comprises a hash value of a block to be migrated; the storage node receiving the block request message processes the request through the data stream, and if the block corresponding to the hash value of the block to be migrated exists locally, the storage node sends the data to the target storage node; if the local value is not present, a null value is returned.
If the reply received by the target storage node is null, the request fails, and the next storage node in the candidate node list is selected to establish connection; and if the target storage node receives the block to be migrated, storing the block to be migrated to the local.
Therefore, when the target storage node needs to be connected with any storage node in the candidate list, after the target storage node is connected with one storage node, the communication flow established by the target storage node and the storage node is maintained, the storage node is used as a neighbor node of the target storage node, and when the candidate node list of any block to be migrated of the target storage block contains the neighbor node, the priority of a block request message of the block to be migrated sent by the target storage block to the neighbor node is highest, so that the time spent by all interactions before the data flow is established can be saved.
In the method, the change of the cluster topology of the storage cluster of the distributed account book is monitored, when one storage node exits, the storage node is timely reselected for backup of the block stored in the exiting storage node, so that the situation that the backup of the data of the block does not meet the requirement and even the data of the block is lost is avoided, when a new storage node is added, the block can be confirmed to be stored by the new storage node and is migrated to the new storage node, the distribution of storage resources can be optimized, and when the cluster topology is changed, the block is only migrated once at most, so that the data migration quantity is smaller.
Referring to fig. 3, a functional block diagram of a digital object migration apparatus according to an embodiment of the present application is shown, where the functional block diagram is applied to a storage cluster of a distributed ledger, and the apparatus includes:
a monitoring module 100, configured to monitor a change of a cluster topology of the storage cluster, where the storage cluster includes a plurality of storage nodes, and the cluster topology includes all online nodes of the storage cluster currently;
the first target storage node determining module 200 is configured to determine, when it is monitored that a change in cluster topology of the storage cluster is that any storage node exits, a respective target storage node for all blocks stored in the storage node;
a second target storage node determining module 300, configured to determine, when it is monitored that a change in a cluster topology of the storage cluster is to join a new storage node, whether the new storage node is a target storage node of any block;
and the migration module 400 is configured to execute a data migration policy by the target storage node, and obtain a block corresponding to the target storage node for storage, where any block is migrated at most once when the data migration policy is executed.
Optionally, the monitoring module includes:
an exit request monitoring unit, configured to monitor an exit request of any node in a cluster topology of the storage cluster;
the node storage monitoring unit is used for sending heartbeat information to a system chain of the distributed account book according to a preset time interval by any storage node in the storage cluster, and adding 1 to the survival identification of the storage node on the system chain; each time a block is produced, determining the respective survival identifications of all storage nodes on the system chain, and subtracting 1 from the survival identifications of all storage nodes; and when the survival identification of any storage node on the system chain is 0 during each block generation, rejecting the storage node in the cluster topology of the storage cluster.
Optionally, the apparatus further comprises:
the version updating module is used for updating the latest version number of the cluster topology when the cluster topology of the storage cluster changes;
the version storage module is used for locally storing the version number of the cluster topology when any storage node in the storage cluster exits;
the comparison module is used for comparing the version number of the cluster topology locally stored in the storage node with the latest version number of the cluster topology when the storage node joins the storage cluster again;
The first scanning module is used for determining whether the storage node is a target storage node of any block or not from the block scanning offset of the local storage when the version number of the local storage cluster topology is the same as the latest version number of the cluster topology;
and the second scanning module is used for scanning all the blocks from 0 when the version number of the locally stored cluster topology is smaller than the latest version number of the cluster topology, and determining whether the storage node is a target storage node of any block or not.
Optionally, the apparatus further comprises:
the first storage layer dividing module is used for dividing a plurality of storage nodes of the storage cluster into a plurality of storage layers;
the first target storage node determining module includes:
and the first target storage node determining unit is used for determining, when any storage node exits, for each storage node in the storage layer to which the storage node belongs, a block hash of each block stored in the storage node and hash values of node identifiers of all the storage nodes in the storage layer, and determining whether the storage node is the target storage node of the block according to the ordering of the hash values.
Optionally, the device further comprises
The second storage layer dividing module is used for dividing a plurality of storage nodes of the storage cluster into a plurality of storage layers;
the first target storage node determining module includes:
a second target storage node determining unit, configured to scan all blocks from 0 by using the new storage node, and obtain each target block stored in a storage layer to which the new storage node belongs; and the new storage node determines the block hash of each target block and the hash value of the node identification of all the storage nodes in the storage layer to which the new storage node belongs, and determines whether the new storage node is the target storage node of the target block according to the ordering of the hash values.
Optionally, the migration module includes:
a candidate node list determining unit, configured to determine a candidate node list of a block to be migrated by using the target storage node, where the candidate node list of the block to be migrated includes a plurality of storage nodes storing the block to be migrated;
a block request message sending unit, configured to select one storage node in the candidate node list by using the target storage node to send a block request message of a block to be migrated; and if the response of the block request message is not received, the target storage node selects the next storage node in the candidate node list to send the block request message of the block to be migrated.
Optionally, the migration module further includes:
and the broadcast searching unit is used for carrying out broadcast searching on all the storage nodes in the candidate node list until all the storage nodes in the candidate node list send block request messages of the blocks to be migrated and do not receive responses of the block request messages.
Optionally, the block request message sending unit further includes:
a communication flow establishing subunit, configured to select one storage node in the candidate node list by using the target storage node, and establish a communication flow with the storage node;
and the communication flow maintenance subunit is used for maintaining the communication flow established by the target storage node and the storage node, taking the storage node as a neighbor node of the target storage node, and when the neighbor node is contained in a candidate node list of any block to be migrated of the target storage block, sending a block request message of the block to be migrated to the neighbor node by the target storage block, wherein the priority is highest.
The present application also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the digital object migration method described in the embodiments.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, the present embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present embodiments have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the present application.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The principles and embodiments of the present application are described herein with specific examples, the above examples being provided only to assist in understanding the methods of the present application and their core ideas; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.

Claims (10)

1. A digital object migration method, applied to a storage cluster of a distributed ledger, the method comprising:
monitoring the change of a cluster topology of the storage cluster, wherein the storage cluster comprises a plurality of storage nodes, and the cluster topology comprises all online nodes of the storage cluster at present;
when the change of the cluster topology of the storage cluster is monitored to be any storage node to exit, respectively determining the corresponding target storage nodes for all the blocks stored in the storage node;
when monitoring that the change of the cluster topology of the storage cluster is to join a new storage node, the new storage node determines whether the new storage node is a target storage node of any block or not;
and the target storage node executes a data migration strategy, and acquires a block corresponding to the target storage node for storage, wherein any block is migrated at most once when the data migration strategy is executed.
2. The method of claim 1, wherein listening for a change in a cluster topology of the storage cluster comprises:
monitoring an exit request of any node in a cluster topology of the storage cluster; or alternatively, the first and second heat exchangers may be,
Any storage node in the storage cluster sends heartbeat information to a system chain of the distributed account book according to a preset time interval, and the survival identification of the storage node on the system chain is increased by 1;
each time a block is produced, determining the respective survival identifications of all storage nodes on the system chain, and subtracting 1 from the survival identifications of all storage nodes;
and when the survival identification of any storage node on the system chain is 0 during each block generation, rejecting the storage node in the cluster topology of the storage cluster.
3. The method according to claim 2, wherein the method further comprises:
updating the latest version number of the cluster topology when the cluster topology of the storage cluster changes;
when any storage node in the storage cluster exits, the version number of the cluster topology at the time of exiting is locally stored in the storage node;
when the storage node joins the storage cluster again, comparing the version number of the cluster topology stored locally in the storage node with the latest version number of the cluster topology;
when the version number of the locally stored cluster topology is the same as the latest version number of the cluster topology, the storage node starts from the locally stored block scanning offset to determine whether the storage node is a target storage node of any block or not;
When the version number of the locally stored cluster topology is smaller than the latest version number of the cluster topology, the storage node starts scanning all the blocks from 0 and determines whether the storage node is a target storage node of any block or not.
4. The method according to claim 1, wherein the method further comprises:
dividing a plurality of storage nodes of the storage cluster into a plurality of storage layers;
when monitoring that the change of the cluster topology of the storage cluster is that any storage node exits, determining the corresponding target storage node for all the blocks stored in the storage node respectively comprises the following steps:
when any storage node exits, for each storage node in the storage layer to which the storage node belongs, determining a block hash of each block stored in the storage node and hash values of node identifiers of all storage nodes in the storage layer, and determining whether the block hash value is a target storage node of the block according to the ordering of the hash values.
5. The method according to claim 1, wherein the method further comprises:
dividing a plurality of storage nodes of the storage cluster into a plurality of storage layers;
when monitoring that the change of the cluster topology of the storage cluster is to join a new storage node, the new storage node determines whether the new storage node is a target storage node of any block or not, including:
The new storage node scans all blocks from 0 to acquire each target block stored in a storage layer to which the new storage node belongs;
and the new storage node determines the block hash of each target block and the hash value of the node identification of all the storage nodes in the storage layer to which the new storage node belongs, and determines whether the new storage node is the target storage node of the target block according to the ordering of the hash values.
6. The method of claim 1, wherein the target storage node performs a data migration policy comprising:
the target storage node determines a candidate node list of a block to be migrated, wherein the candidate node list of the block to be migrated comprises a plurality of storage nodes for storing the block to be migrated;
the target storage node selects one storage node in the candidate node list to send a block request message of a block to be migrated;
and if the response of the block request message is not received, the target storage node selects the next storage node in the candidate node list to send the block request message of the block to be migrated.
7. The method of claim 6, wherein the target storage node performs a data migration policy, further comprising:
And the target storage node broadcasts and searches all the storage nodes in the candidate node list for the block to be migrated until the block request message of the block to be migrated is sequentially sent to all the storage nodes in the candidate node list and the response of the block request message is not received.
8. The method of claim 6, wherein the target storage node selecting one storage node in the candidate node list to send a block request message for a block to be migrated, further comprising:
the target storage node selects one storage node in the candidate node list and establishes communication flow with the storage node;
and maintaining the communication flow established between the target storage node and the storage node, taking the storage node as a neighbor node of the target storage node, and when a candidate node list of any block to be migrated of the target storage block contains the neighbor node, sending a block request message of the block to be migrated to the neighbor node by the target storage block with the highest priority.
9. A digital object migration apparatus for use with a storage cluster of a distributed ledger, the apparatus comprising:
The monitoring module is used for monitoring the change of the cluster topology of the storage cluster, wherein the storage cluster comprises a plurality of storage nodes, and the cluster topology comprises all online nodes of the storage cluster at present;
the first target storage node determining module is used for determining the corresponding target storage nodes for all the blocks stored in the storage nodes respectively when monitoring that the change of the cluster topology of the storage cluster is that any storage node exits;
the second target storage node determining module is used for determining whether the new storage node is a target storage node of any block or not when the change of the cluster topology of the storage cluster is monitored to be a new storage node;
and the migration module is used for executing the data migration strategy by the target storage node, acquiring the block corresponding to the target storage node and storing the block, wherein any block is migrated at most once when the data migration strategy is executed.
10. A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, which computer program, when executed by a processor, implements the digital object migration method according to any one of claims 1 to 8.
CN202311264989.2A 2023-09-27 2023-09-27 Digital object migration method, device and storage medium Pending CN117348806A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311264989.2A CN117348806A (en) 2023-09-27 2023-09-27 Digital object migration method, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311264989.2A CN117348806A (en) 2023-09-27 2023-09-27 Digital object migration method, device and storage medium

Publications (1)

Publication Number Publication Date
CN117348806A true CN117348806A (en) 2024-01-05

Family

ID=89370280

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311264989.2A Pending CN117348806A (en) 2023-09-27 2023-09-27 Digital object migration method, device and storage medium

Country Status (1)

Country Link
CN (1) CN117348806A (en)

Similar Documents

Publication Publication Date Title
CN109995850B (en) Block chain system and transaction processing method thereof
CN112737916B (en) Data processing method based on block chain network and related device
JP5227955B2 (en) A distributed hash mechanism for self-regulating networks
CN111190714A (en) Cloud computing task scheduling system and method based on block chain
US20070008880A1 (en) Router redundancy in data communication networks
CN106953767B (en) Method, device and system for networking between devices
KR20110021931A (en) Method of determining a routing path
CN112055048B (en) P2P network communication method and system for high-throughput distributed account book
CN112200681B (en) Service processing method, information processing method and node equipment of block chain network
CN109525678B (en) Block chain network system and corresponding node device discovery method
CN112417001B (en) Data processing method based on block chain network and related equipment
CN113411376A (en) Sensor data processing method and device based on block chain fragmentation storage
CN112202877B (en) Gateway linkage method, gateway, cloud server and user terminal
EP1719325B1 (en) Method for optimally utilizing a peer to peer network
CN110971702A (en) Service calling method and device, computer equipment and storage medium
CN112583712A (en) Block chain router and block chain network
CN110990448B (en) Distributed query method and device supporting fault tolerance
CN110602232A (en) Terminal system version downloading method, device and system based on peer-to-peer network idea
CN111917748B (en) Intelligent laser remote control system and method based on IPFS + alliance chain
CN111800516B (en) Internet of things equipment management method and device based on P2P
CN117348806A (en) Digital object migration method, device and storage medium
WO2018107480A1 (en) Service scheduling method and system
CN112583572B (en) Block chain network, service processing method, device and equipment
CN114255020A (en) Block chain transaction processing method, device, equipment and storage medium based on multiple chains
CN114465933A (en) Block chain network transmission method and transmission medium based on KAD (Kad Gemini) model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination