CN117492644A - Data migration method and device, electronic equipment and storage medium - Google Patents
Data migration method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117492644A CN117492644A CN202311436436.0A CN202311436436A CN117492644A CN 117492644 A CN117492644 A CN 117492644A CN 202311436436 A CN202311436436 A CN 202311436436A CN 117492644 A CN117492644 A CN 117492644A
- Authority
- CN
- China
- Prior art keywords
- data
- storage node
- identifier
- pool
- data pool
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 95
- 230000005012 migration Effects 0.000 title claims abstract description 95
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000013507 mapping Methods 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims description 21
- 238000013500 data storage Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 abstract description 26
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000004590 computer program Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001680 brushing effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data migration method, a data migration device, electronic equipment and a storage medium, relates to the technical field of computers, and solves the technical problems that due to the fact that the data size in the data migration process is large, the workload of related staff in the process of manually carrying out data migration is large, data migration errors can be caused, and the accuracy and the effectiveness of data migration are affected. The method comprises the following steps: determining a first identifier and a second identifier, wherein the first identifier is the identifier of the first storage node, and the second identifier is the identifier of the second data pool; performing preset algorithm operation on the second identifier and a third identifier to obtain a target identifier, wherein the third identifier is obtained by performing hash mapping operation on the first identifier; and determining the storage node corresponding to the target identifier as a second storage node.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data migration method, a data migration device, an electronic device, and a storage medium.
Background
Currently, during data migration, related personnel are required to copy data in an old data pool to a new data pool, and restart the server to complete the data migration.
However, in the above method, because the data volume is larger in the data migration process, the workload of the related staff is larger in the process of manually performing the data migration, which may cause data migration errors and affect the accuracy and effectiveness of the data migration.
Disclosure of Invention
The application provides a data migration method, a data migration device, electronic equipment and a storage medium, which solve the technical problems that due to the fact that the data volume in the data migration process is large, the workload of related staff in the process of manually carrying out data migration is large, data migration errors can be caused, and the accuracy and the effectiveness of data migration are affected.
In a first aspect, the present application provides a data migration method, including: acquiring a plurality of data, wherein the plurality of data are data included in a first storage node, and the first storage node is a storage node included in the first data pool; determining a second storage node, the second storage node being a storage node included in a second data pool; the plurality of data is stored to the second storage node such that the plurality of data is migrated from the first data pool to the second data pool.
Optionally, the determining the second storage node includes: determining a first identifier and a second identifier, wherein the first identifier is the identifier of the first storage node, and the second identifier is the identifier of the second data pool; performing preset algorithm operation on the second identifier and a third identifier to obtain a target identifier, wherein the third identifier is obtained by performing hash mapping operation on the first identifier; and determining the storage node corresponding to the target identifier as a second storage node.
In the application, the electronic device can determine the first identifier and the second identifier, hash mapping operation is performed on the first identifier to obtain the third identifier, and then the electronic device performs preset algorithm operation on the second identifier and the third identifier, so that the target identifier can be accurately and conveniently obtained.
Optionally, the data migration method may further include: and deleting the first storage node from the first data pool in case that the plurality of data are successfully stored in the second storage node.
In this application, when the plurality of data is successfully stored in the second storage node, it is described that the plurality of data is successfully migrated from the first data pool to the second data pool, and at this time, the electronic device may effectively delete the first storage node from the first data pool, so as to accurately and effectively delete the successfully migrated data from the first data pool.
Optionally, the data migration method may further include: receiving a data reading request of a target user, wherein the data reading request is used for requesting to read target data from the first storage node, and the target data is one of the plurality of data; determining whether the first storage node exists in the first data pool; determining if the second storage node is present in the second data pool if the first storage node is not present in the first data pool; and in the case where the second storage node exists in the second data pool, transmitting the target data to the target user based on the plurality of data included in the second storage node.
In the application, after the first data pool deletes the migrated data, in order not to affect related operations of the user side, the request of the target user may be sent to the first data pool, after the electronic device receives the data reading request of the target user, it is first determined whether there is a first storage node in the first data pool, where the first storage node does not exist in the first data pool, which indicates that the first storage node may have been migrated to the second data pool successfully, at this time, the electronic device may determine whether there is the second storage node in the second data pool, where the second storage node exists in the second data pool, which indicates that the data in the first storage node has been successfully stored to the second storage node, and at this time, the electronic device may accurately and effectively send the target data to the target user based on a plurality of data included in the second storage node, thereby implementing data migration without perception of the target user, and improving use experience of the user.
Optionally, the data migration method may further include: and under the condition that a plurality of storage nodes included in the first data pool are stored in the second data pool, determining the second data pool as a target data pool, wherein the target data pool is a data pool for a preset operation of the target user, and the preset operation comprises data reading and data storage.
In the application, under the condition that the plurality of storage nodes included in the first data pool are all stored in the second data pool, the fact that all data included in the first data pool are migrated into the second data pool is explained, and at the moment, the electronic equipment can accurately and effectively determine the second data pool as a target data pool, so that migration of the data is accurately and effectively achieved under the condition that a user does not feel.
In a second aspect, the present application provides a data migration apparatus, including: the device comprises an acquisition module, a determination module and a storage module; the acquisition module is used for acquiring a plurality of data, wherein the plurality of data are data included in a first storage node, and the first storage node is a storage node included in the first data pool; the determining module is configured to determine a second storage node, where the second storage node is a storage node included in a second data pool; the storage module is used for storing the data to the second storage node so as to enable the data to be migrated from the first data pool to the second data pool.
Optionally, the data migration device further includes a processing module; the determining module is further configured to determine a first identifier and a second identifier, where the first identifier is an identifier of the first storage node, and the second identifier is an identifier of the second data pool; the operation module is used for carrying out preset algorithm operation on the second identifier and a third identifier to obtain a target identifier, wherein the third identifier is obtained by carrying out hash mapping operation on the first identifier; the determining module is further configured to determine a storage node corresponding to the target identifier as a second storage node.
Optionally, the data migration device further includes a processing module; the processing module is configured to delete the first storage node from the first data pool if the plurality of data is successfully stored in the second storage node.
Optionally, the data migration device further includes a receiving module and a sending module; the receiving module is used for receiving a data reading request of a target user, wherein the data reading request is used for requesting to read target data from the first storage node, and the target data is one of the data; the determining module is further configured to determine whether the first storage node exists in the first data pool; the determining module is further configured to determine, if the second storage node exists in the second data pool, in a case where the first storage node does not exist in the first data pool; the sending module is configured to send, in a case where the second storage node exists in the second data pool, the target data to the target user based on the plurality of data included in the second storage node.
Optionally, the determining module is further configured to determine, in a case where a plurality of storage nodes included in the first data pool are all stored in the second data pool, the second data pool as a target data pool, where the target data pool is a data pool for performing a preset operation for the target user, where the preset operation includes data reading and data storing.
In a third aspect, the present application provides an electronic device, comprising: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional data migration methods of the first aspect described above.
In a fourth aspect, the present application provides a computer readable storage medium having instructions stored thereon which, when executed by an electronic device, enable the electronic device to perform any one of the above-described optional data migration methods of the first aspect.
According to the data migration method, the device, the electronic equipment and the storage medium, the electronic equipment firstly obtains a plurality of data which are included in a first storage structure of a first data pool and need to be subjected to data migration, then the electronic equipment determines a second storage structure of a second data pool, and at the moment, the electronic equipment stores the plurality of data to the second storage node so that the plurality of data are migrated from the first data pool to the second data pool. In the application, the electronic device can effectively determine a plurality of data to be subjected to data migration based on the data object set, acquire the plurality of data from the first storage node in the first data pool, then determine the second storage node in the second data pool, and store the plurality of data into the second storage node, so that the migration of the plurality of data from the first data pool to the second data pool is effectively and accurately realized, and the accuracy and the high efficiency of data migration are improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating another data migration method according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of a process for constructing a target cache structure according to an embodiment of the present application;
fig. 4 is a schematic diagram of a process of performing data migration in a target cache structure according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a specific process for performing data migration from a first data pool to a second data pool according to an embodiment of the present disclosure;
FIG. 6 is a flowchart illustrating another data migration method according to an embodiment of the present disclosure;
FIG. 7 is a schematic diagram of a process for deleting a first storage node from a first data pool according to an embodiment of the present application;
FIG. 8 is a flowchart illustrating another data migration method according to an embodiment of the present disclosure;
fig. 9 is a schematic diagram of a processing procedure of a data read request for a target user according to an embodiment of the present application;
FIG. 10 is a schematic diagram of a processing procedure of a data write request to a target user according to an embodiment of the present application;
FIG. 11 is a flowchart illustrating another data migration method according to an embodiment of the present disclosure;
FIG. 12 is a schematic diagram of a process for migrating a data pool according to an embodiment of the present application;
fig. 13 is a schematic structural diagram of a data migration device according to an embodiment of the present application;
fig. 14 is a schematic structural diagram of another data migration apparatus according to an embodiment of the present application.
Detailed Description
The data migration method, the device, the electronic equipment and the storage medium provided by the embodiment of the application are described in detail below with reference to the accompanying drawings.
The terms "first" and "second" and the like in the description and in the drawings are used for distinguishing between different objects and not for describing a particular sequence of objects, e.g., a first storage node and a second storage node and the like are used for distinguishing between different storage nodes and not for describing a particular sequence of storage nodes.
Furthermore, references to the terms "comprising" and "having" and any variations thereof in the description of the present application are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed but may optionally include other steps or elements not listed or inherent to such process, method, article, or apparatus.
It should be noted that, in the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
In the description of the present application, unless otherwise indicated, the meaning of "a plurality" means two or more.
Some concepts related to a data migration method, a device, an electronic apparatus, and a storage medium provided in the embodiments of the present application are explained below.
Ceph: the distributed storage system provides distributed object, file and block storage service for users and is used in the fields of big data and cloud computing.
Cephfs: ceph provides an upper level application of a distributed file system, user file data is stored in a data pool, and a file system can have multiple data pools.
MDS: the service in Cephfs is used to manage file system metadata.
Based on the description in the background art, because in the related art, the data volume in the data migration process is larger, the workload of related staff in the process of manually carrying out data migration is larger, the data migration error can be caused, and the accuracy and the effectiveness of the data migration are affected. Based on this, the embodiment of the application provides a data migration method, a device, an electronic device and a storage medium, where the electronic device may first determine, based on a data object set, the plurality of data to be migrated effectively, and then acquire the plurality of data from a first storage node in the first data pool, then determine a second storage node in the second data pool, and store the plurality of data in the second storage node, so as to effectively and accurately migrate the plurality of data from the first data pool to the second data pool, thereby improving accuracy and efficiency of data migration.
By way of example, the electronic device may be a cell phone, tablet, desktop, laptop, handheld computer, notebook, ultra-mobile personal computer, UMPC, netbook, and cellular phone, personal digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR) \virtual reality (VR) device, and the specific form of the electronic device is not particularly limited in embodiments of the present application. The system can perform man-machine interaction with a user through one or more modes of a keyboard, a touch pad, a touch screen, a remote controller, voice interaction or handwriting equipment and the like.
Optionally, the electronic device may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, network acceleration services (content delivery network, CDN), basic cloud computing services such as big data and artificial intelligence platforms, and the like.
As shown in fig. 1, the data migration method provided in the embodiment of the present application may include S101-S103.
S101, the electronic equipment acquires a plurality of data.
The plurality of data are data included in a first storage node, and the first storage node is a storage node included in the first data pool.
In one implementation manner of the embodiment of the present application, in the process of cluster service, an old server often needs to be taken off-shelf for maintenance, and when a storage node where one data pool is located needs to be taken off-shelf, data in the data pool needs to be migrated to another data pool.
In connection with the above embodiments, it may be appreciated that the first data pool is an old data pool (i.e., a data pool that needs to migrate data in the data pool to another data pool).
Optionally, the electronic device may determine a data object set (PG), where the PG set is a set of data objects that need to undergo data migration, and the electronic device may determine, according to an object list included in the PG set, a plurality of data slices corresponding to the object list, and then perform data slice searching according to the PG set, so as to obtain data included in the plurality of data slices.
Optionally, the electronic device may acquire the plurality of data according to an acquisition mode corresponding to a storage engine storing the plurality of data.
By way of example, the storage engine may be a file store (filestore) storage engine that corresponds to files in the local file system xfs, so that the electronic device's acquisition is to read the xfs file's data.
S102, the electronic device determines a second storage node.
Wherein the second storage node is a storage node included in a second data pool.
It is understood that the second storage node is a storage node in the second data pool for storing data included in the first storage node.
In connection with the above embodiments, it may be understood that the second data pool is a new data pool, and data in the old data pool needs to be migrated to the new data pool, that is, data in a storage node included in the first data pool needs to be migrated to the second data pool.
In some embodiments, as shown in fig. 2 in conjunction with fig. 1, the determining the second storage node may specifically include S1021-S1023.
S1021, the electronic device determines the first identifier and the second identifier.
Wherein the first identifier is an identifier of the first storage node, and the second identifier is an identifier of the second data pool.
In one implementation manner of the embodiment of the present application, the electronic device may determine an identifier of a first storage node in the first data pool that needs to be migrated, and an identifier of the second data pool.
S1022, the electronic device performs preset algorithm operation on the second identifier and the third identifier to obtain the target identifier.
The third identifier is obtained by performing hash mapping operation on the first identifier.
It can be understood that the electronic device first performs hash mapping operation on the first identifier to obtain a third identifier, and then performs preset algorithm operation on the second identifier and the third identifier to obtain a target identifier.
The preset algorithm operation may be, for example, a credit algorithm.
S1023, the electronic equipment determines the storage node corresponding to the target identifier as a second storage node.
In one implementation manner of the embodiment of the present application, the electronic device may determine, based on the target identifier, a second storage node and at least one copy node of the second storage node, and when the second storage node is abnormal, may replace the second storage node with the at least one copy node.
In the embodiment of the application, the electronic device can determine the first identifier and the second identifier, hash mapping operation is performed on the first identifier to obtain the third identifier, and then the electronic device performs preset algorithm operation on the second identifier and the third identifier, so that the target identifier can be accurately and conveniently obtained.
S103, the electronic device stores the plurality of data to the second storage node so that the plurality of data are migrated from the first data pool to the second data pool.
In combination with the foregoing embodiment, it may be appreciated that the electronic device may store the plurality of data in the second storage node and at least one copy node of the second storage node, so as to avoid losing data when there is an anomaly in the second storage node.
The technical scheme provided by the embodiment at least has the following beneficial effects: S101-S103 can be seen: the electronic equipment firstly acquires a plurality of data which are included in a first storage structure and need to be subjected to data migration in a first data pool, then the electronic equipment determines a second storage structure in the second data pool, and at the moment, the electronic equipment stores the plurality of data to the second storage node so as to enable the plurality of data to be migrated from the first data pool to the second data pool. According to the method and the device for data migration, the electronic device can effectively determine the plurality of data needing to be subjected to data migration based on the data object set, further acquire the plurality of data from the first storage node in the first data pool, then determine the second storage node in the second data pool, and store the plurality of data into the second storage node, so that the plurality of data are effectively and accurately migrated from the first data pool to the second data pool, and accuracy and high efficiency of data migration are improved.
In one implementation manner of the embodiment of the present application, the electronic device may construct the first data pool and the second data pool into a target cache structure, where the first data pool is a cache layer, the second data pool is a base layer, and in the cache structure, multiple storage nodes in the first data pool copy data from the first data pool to the second data pool in a manner of brushing down.
Specifically, the electronic device may send a build cache structure indication to a master monitor node included in the monitor cluster, where the build cache structure indication includes the first data pool and the second data pool, and where the master monitor and a slave monitor included in the monitor cluster may send the build cache structure indication to a plurality of storage nodes included in the first data pool and a plurality of storage nodes included in the second data pool, so that the first data pool and the second data pool are built into a target cache structure, and the target cache structure migrates data from the first data pool to the second data pool.
The following describes, in an example, a process of constructing a target cache structure in an embodiment of the present application.
For example, as shown in fig. 3, a build cache structure indication is first sent to the monitor cluster, then the build cache structure indication is synchronized by the master monitor to the slave monitor, then the build cache structure indication is sent by the monitor cluster to a plurality of storage nodes included in the first data pool and a plurality of storage nodes included in the second data pool, such that the first data pool and the second data pool are built as a target cache structure, such that the first data pool perceives the second data pool and data is migrated by the first data pool to the second data pool.
The following describes, in an example, a process of data migration by using a target cache structure in an embodiment of the present application.
Illustratively, as shown in fig. 4, the target cache structure copies data from the first data pool to the second data pool through a lower flush, at the same time, the business service may not sense to continue to read and write the data of the/data subtree distributed to the first data pool, and if the data requested to be read and written in the read and write request is not already in the first data pool, the read and write request may be proxied (or forwarded) to the second data pool for processing.
In one implementation of the embodiment of the present application, file data in Cephfs is fragmented in units of 4M and dispersed among a plurality of storage nodes included in the first data pool.
The following describes, in an example, a specific process of data migration from the first data pool to the second data pool in an embodiment of the present application.
As shown in fig. 5, the method includes determining, according to the PG set and the data lookup module, the shard (i.e., the shard 1 and the shard 4) of the data to be acquired, where the file data in Cephfs is partitioned in units of 4M and is dispersed in a plurality of storage nodes included in the first data pool, where the shard 1 and the shard 4 are both in the first storage node of the first data pool, locating, by the data locating module, the second storage node (including a master storage node and two slave storage nodes) in the second data pool after the data in the shard 1 and the shard 4 are acquired, then storing the acquired data in the master storage structure, and performing copy synchronization to the slave storage structure by the master storage structure.
Referring to fig. 1, as shown in fig. 6, the data migration method in the embodiment of the present application may further include S104.
And S104, under the condition that a plurality of data are successfully stored in the second storage node, the electronic equipment deletes the first storage node from the first data pool.
It should be appreciated that in the event that the plurality of data is successfully stored to the second storage node, indicating that the plurality of data was successfully migrated from the first data pool to the second data pool, the electronic device may effectively delete the first storage node from the first data pool such that the successfully migrated data is accurately and effectively deleted from the first data pool.
The process of deleting the first storage node from the first data pool in the embodiment of the present application is described below as an example.
Illustratively, as shown in fig. 7, in the case that the data in the tile 1 is successfully stored to the master storage node and the slave storage node of the second data pool through the write request, the second data pool sends a storage completion response to the first data pool, and the first data pool performs an eviction callback (i.e., delete) after receiving the storage completion response, i.e., deletes the tile 1 from the first data pool to complete data eviction, so that after migrating the data tile, the first data pool is cleaned up.
Referring to fig. 1, as shown in fig. 8, the data migration method in the embodiment of the present application may further include S105 to S108.
S105, the electronic equipment receives a data reading request of the target user.
The data reading request is used for requesting to read target data from the first storage node, wherein the target data is one of the plurality of data.
It will be appreciated that after the first data pool deletes the migrated data, the request of the target user will be sent to the first data pool in order not to affect the related operation of the user side.
S106, the electronic device determines whether a first storage node exists in the first data pool.
In one implementation manner of the embodiment of the present application, during the data migration process, the target user sends a data reading request, where the electronic device may first determine whether a storage node (i.e., a first storage node) required by the request of the target user exists in the first data pool.
And S107, under the condition that the first storage node does not exist in the first data pool, the electronic equipment determines whether a second storage node exists in the second data pool.
It should be appreciated that in the event that the first storage node is not present in the first data pool, indicating that the first storage node may have successfully migrated to the second data pool, the electronic device may determine whether the second storage node is present in the second data pool.
It will be appreciated that the data stored in the second storage node is data migrated from the first storage node, i.e. the second storage node is identical to the first storage node.
And S108, in the case that the second storage node exists in the second data pool, the electronic equipment sends target data to the target user based on the plurality of data included in the second storage node.
It should be appreciated that in the case where the second storage node is present in the second data pool, indicating that the data in the first storage node has been successfully stored to the second storage node, the electronic device may send target data to the target user based on the plurality of data included in the second storage node.
In this embodiment of the present invention, after the first data pool deletes migrated data, in order not to affect related operations of a user side, after receiving a data reading request of the target user, the electronic device may determine whether a first storage node exists in the first data pool, and in the case that the first storage node does not exist in the first data pool, it is indicated that the first storage node may have been migrated to the second data pool successfully, at this time, the electronic device may determine whether the second storage node exists in the second data pool, in the case that the second storage node exists in the second data pool, it is indicated that data in the first storage node has been successfully stored to the second storage node, and at this time, the electronic device may accurately and effectively send the target data to the target user based on a plurality of data included in the second storage node, thereby implementing data migration under the condition that the target user does not have awareness, and improving use experience of the user.
The following describes, as an example, a processing procedure of a data read request to the target user in the embodiment of the present application.
As shown in fig. 9, when the user read request 1 is received, the slice 2 corresponding to the user read request 1 is searched in the first data pool, and the data in the slice 2 is returned to the user, when the user read request 2 is received, the slice 4 corresponding to the user read request 2 is searched in the first data pool, at this time, the slice 4 is found to be evicted by the first data pool, at this time, the user read request 2 is forwarded to the second data pool, and the slice 4 is searched in the second data pool, and the data in the slice 4 is returned to the user.
In an implementation manner of this embodiment of the present application, when the request sent by the target user is a data write request (i.e. a data storage request), in order not to affect a related operation of a user side, the request of the target user may continue to be sent to the first data pool, after receiving the data write request of the target user, the electronic device first determines whether a first storage node exists in the first data pool, if the first storage node does not exist in the first data pool, it indicates that the first storage node may have been migrated to the second data pool successfully, at this time, the electronic device may determine whether the second storage node exists in the second data pool, if the second storage node exists in the second data pool, it indicates that data in the first storage node has been successfully stored to the second storage node, at this time, based on the data write request of the target user, modify the data included in the second storage node, if the first storage node has not been deleted, and if the first storage node has not been migrated, it indicates that the first storage node has not been migrated to the second storage node, and then the electronic device may store the modified data in the second storage pool after the second storage node has been updated based on the data write request of the target user.
The processing procedure of the data write request to the target user in the embodiment of the present application is described below as an example.
As shown in fig. 10, when the user write request 1 is received, a slice 2 corresponding to the user read request 1 is searched in the first data pool, the data in the slice 2 is modified according to the user write request 1, then the modified slice 2 is data-migrated, that is, the modified slice 2 in the first data pool is migrated to the second data pool, then the modified slice 2 in the first data pool is evicted, when the user write request 2 is received, a slice 4 corresponding to the user read request 2 is searched in the first data pool, at this time, the slice 4 is found to be evicted by the first data pool, at this time, the user write request 2 is forwarded to the second data pool, the slice 4 is searched in the second data pool, and the data in the slice 4 is modified according to the user write request 2.
Referring to fig. 1, as shown in fig. 11, the data migration method in the embodiment of the present application may further include S109.
And S109, in the case that a plurality of storage nodes included in the first data pool are stored in the second data pool, the electronic equipment determines the second data pool as a target data pool.
The target data pool is a data pool for performing a preset operation by the target user, wherein the preset operation comprises data reading and data storage.
It can be understood that the target data pool is a data pool for the user side to request to send after data migration.
It should be understood that, in the case where the plurality of storage nodes included in the first data pool are all stored in the second data pool, it is explained that the data included in the first data pool has been migrated to the second data pool, at this time, the electronic device may accurately and effectively determine the second data pool as the target data pool, so that the migration of the data is accurately and effectively achieved without perception of the user.
In one implementation manner of the embodiment of the present application, after all data in the first data pool is flushed, that is, the first data pool is already emptied, the directory subtree needs to be switched in pool distribution at this time, that is, the first data pool is changed to the second data pool, which is implemented by topology updating and metadata synchronization of the storage node.
The following describes, in one example, a process of migrating a data pool in an embodiment of the present application.
Exemplary, as shown in fig. 12, when the first data pool is empty, the storage node is updated based on the monitor cluster, then the directory subtree is switched in pool distribution, that is, the second data pool is determined as the target data pool, at this time, the second data pool replaces the first data pool, and then the index node is updated to the client, so that the client switches the read-write request from the first data pool to the second data pool, thereby completing the migration of the data pools.
The embodiment of the application may divide the functional modules of the electronic device according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated modules may be implemented in hardware or in software functional modules. It should be noted that, in the embodiment of the present application, the division of the modules is schematic, which is merely a logic function division, and other division manners may be implemented in actual implementation.
In the case of dividing the respective functional modules with the respective functions, fig. 13 shows a schematic diagram of one possible configuration of the data migration apparatus involved in the above-described embodiment, and as shown in fig. 13, the data migration apparatus 10 may include: an acquisition module 101, a determination module 102 and a storage module 103.
The acquiring module 101 is configured to acquire a plurality of data, where the plurality of data is data included in a first storage node, and the first storage node is a storage node included in the first data pool.
A determining module 102, configured to determine a second storage node, where the second storage node is a storage node included in the second data pool.
A storage module 103, configured to store the plurality of data to the second storage node, so that the plurality of data is migrated from the first data pool to the second data pool.
Optionally, the data migration apparatus 10 further includes a processing module 104.
The determining module 102 is further configured to determine a first identifier, which is an identifier of the first storage node, and a second identifier, which is an identifier of the second data pool.
And the processing module 104 is configured to perform a preset algorithm operation on the second identifier and a third identifier to obtain a target identifier, where the third identifier is obtained by performing a hash mapping operation on the first identifier.
The determining module 102 is further configured to determine a storage node corresponding to the target identifier as a second storage node.
Optionally, the data migration apparatus 10 further includes a processing module 104.
And a processing module 104, configured to delete the first storage node from the first data pool if the plurality of data is successfully stored in the second storage node.
Optionally, the data migration apparatus 10 further includes a receiving module 105 and a sending module 106.
A receiving module 105, configured to receive a data read request of a target user, where the data read request is used to request to read target data from the first storage node, and the target data is one of the plurality of data.
The determining module 102 is further configured to determine whether the first storage node exists in the first data pool.
The determining module 102 is further configured to determine whether the second storage node exists in the second data pool in a case where the first storage node does not exist in the first data pool.
And a sending module 106, configured to send the target data to the target user based on the plurality of data included in the second storage node when the second storage node exists in the second data pool.
Optionally, the determining module 102 is further configured to determine, in a case where a plurality of storage nodes included in the first data pool are all stored in the second data pool, the second data pool as a target data pool, where the target data pool is a data pool for performing a preset operation for the target user, where the preset operation includes data reading and data storing.
In the case of an integrated unit, fig. 14 shows a schematic diagram of a possible configuration of the data migration apparatus involved in the above-described embodiment. As shown in fig. 14, the data migration apparatus 20 may include: a processing module 201 and a communication module 202. The processing module 201 may be configured to control and manage the actions of the data migration apparatus 20. The communication module 202 may be used to support communication of the data migration apparatus 20 with other entities. Optionally, as shown in fig. 14, the data migration apparatus 20 may further include a storage module 203 for storing program codes and data of the data migration apparatus 20.
Wherein the processing module 201 may be a processor or a controller. The communication module 202 may be a transceiver, a transceiver circuit, a communication interface, or the like. The storage module 203 may be a memory.
When the processing module 201 is a processor, the communication module 202 is a transceiver, and the storage module 203 is a memory, the processor, the transceiver, and the memory may be connected through a bus. The bus may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, or the like. The buses may be divided into address buses, data buses, control buses, etc.
It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber terminal line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more servers, data centers, etc. that can be integrated with the medium. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.
The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (12)
1. A method of data migration, the method comprising:
acquiring a plurality of data, wherein the plurality of data are data included in a first storage node, and the first storage node is a storage node included in a first data pool;
determining a second storage node, wherein the second storage node is a storage node included in a second data pool;
storing the plurality of data to the second storage node such that the plurality of data is migrated from the first data pool to the second data pool.
2. The data migration method of claim 1, wherein the determining the second storage node comprises:
determining a first identifier and a second identifier, wherein the first identifier is the identifier of the first storage node, and the second identifier is the identifier of the second data pool;
Performing preset algorithm operation on the second identifier and a third identifier to obtain a target identifier, wherein the third identifier is obtained by performing hash mapping operation on the first identifier;
and determining the storage node corresponding to the target identifier as a second storage node.
3. The data migration method of claim 1, further comprising:
and deleting the first storage node from the first data pool under the condition that the plurality of data are successfully stored in the second storage node.
4. The data migration method of claim 1, further comprising:
receiving a data reading request of a target user, wherein the data reading request is used for requesting to read target data from the first storage node, and the target data is one of the plurality of data;
determining whether the first storage node exists in the first data pool;
determining if the second storage node is present in the second data pool if the first storage node is not present in the first data pool;
and transmitting the target data to the target user based on the plurality of data included in the second storage node in the case where the second storage node exists in the second data pool.
5. The data migration method of any one of claims 1-4, wherein the method further comprises:
and under the condition that a plurality of storage nodes included in the first data pool are stored in the second data pool, determining the second data pool as a target data pool, wherein the target data pool is a data pool for a target user to perform preset operation, and the preset operation comprises data reading and data storage.
6. A data migration apparatus, comprising: the device comprises an acquisition module, a determination module and a storage module;
the acquisition module is used for acquiring a plurality of data, wherein the plurality of data are data included in a first storage node, and the first storage node is a storage node included in a first data pool;
the determining module is configured to determine a second storage node, where the second storage node is a storage node included in a second data pool;
the storage module is configured to store the plurality of data to the second storage node, so that the plurality of data is migrated from the first data pool to the second data pool.
7. The data migration apparatus of claim 6, further comprising a processing module;
The determining module is further configured to determine a first identifier and a second identifier, where the first identifier is an identifier of the first storage node, and the second identifier is an identifier of the second data pool;
the processing module is used for carrying out preset algorithm operation on the second identifier and a third identifier to obtain a target identifier, wherein the third identifier is obtained by carrying out hash mapping operation on the first identifier;
the determining module is further configured to determine a storage node corresponding to the target identifier as a second storage node.
8. The data migration apparatus of claim 6, further comprising a processing module;
the processing module is configured to delete the first storage node from the first data pool if the plurality of data is successfully stored in the second storage node.
9. The data migration apparatus of claim 6, further comprising a receiving module and a transmitting module;
the receiving module is used for receiving a data reading request of a target user, wherein the data reading request is used for requesting to read target data from the first storage node, and the target data is one of the plurality of data;
The determining module is further configured to determine whether the first storage node exists in the first data pool;
the determining module is further configured to determine, if the second storage node exists in the second data pool, in a case where the first storage node does not exist in the first data pool;
the sending module is configured to send, in a case where the second storage node exists in the second data pool, the target data to the target user based on the plurality of data included in the second storage node.
10. The data migration apparatus according to any one of claims 6 to 9, wherein
The determining module is further configured to determine, when a plurality of storage nodes included in the first data pool are all stored in the second data pool, the second data pool as a target data pool, where the target data pool is a data pool where a preset operation is performed by the target user, and the preset operation includes data reading and data storage.
11. An electronic device, the electronic device comprising:
a processor;
a memory configured to store the processor-executable instructions;
Wherein the processor is configured to execute the instructions to implement the data migration method of any one of claims 1-5.
12. A computer readable storage medium having instructions stored thereon, which, when executed by an electronic device, cause the electronic device to perform the data migration method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311436436.0A CN117492644A (en) | 2023-10-31 | 2023-10-31 | Data migration method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311436436.0A CN117492644A (en) | 2023-10-31 | 2023-10-31 | Data migration method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117492644A true CN117492644A (en) | 2024-02-02 |
Family
ID=89684099
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311436436.0A Pending CN117492644A (en) | 2023-10-31 | 2023-10-31 | Data migration method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117492644A (en) |
-
2023
- 2023-10-31 CN CN202311436436.0A patent/CN117492644A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8510499B1 (en) | Solid state drive caching using memory structures to determine a storage space replacement candidate | |
| JP5727020B2 (en) | Cloud computing system and data synchronization method thereof | |
| CN103067433B (en) | A kind of data migration method of distributed memory system, equipment and system | |
| US7673112B2 (en) | Volume management system and method | |
| US8386717B1 (en) | Method and apparatus to free up cache memory space with a pseudo least recently used scheme | |
| CN111158851B (en) | Rapid deployment method of virtual machine | |
| KR20210040864A (en) | File directory traversal method, apparatus, device, and medium | |
| CN110413568A (en) | A kind of data multiplexing method, device, electronic equipment and storage medium | |
| CN115801699A (en) | CDN scheduling method, device and system | |
| CN111147312B (en) | Resource allocation management method and device, resource allocation cache management method and device, and allocation management system | |
| CN112015696A (en) | Data access method, data relationship setting method, data access device, data relationship setting device and storage medium | |
| EP4295235A1 (en) | Cache indexing using data addresses based on data fingerprints | |
| US20230161664A1 (en) | Method of responding to operation, electronic device, and storage medium | |
| KR102665749B1 (en) | Method and apparatus for ensuring continued device operational reliability in cloud-degraded mode | |
| KR101419275B1 (en) | Data synchronizing and servicing apparatus and method based on cloud storage | |
| CN115827778A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN117492644A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN114610740B (en) | Data version management method and device of medical data platform | |
| CN114422538B (en) | Multi-cloud storage system, multi-cloud data reading and writing method and electronic equipment | |
| CN113806013B (en) | Storage link cleaning method, device, equipment and medium | |
| CN115390754A (en) | Hard disk management method and device | |
| CN115587119A (en) | Database query method and device, electronic equipment and storage medium | |
| WO2014054173A1 (en) | Distributed data management system and program | |
| CN110058790B (en) | Method, apparatus and computer program product for storing data | |
| CN117370295A (en) | File copying method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |