CN116860166A - Data migration method, device, electronic equipment and storage medium - Google Patents
Data migration method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116860166A CN116860166A CN202310816417.4A CN202310816417A CN116860166A CN 116860166 A CN116860166 A CN 116860166A CN 202310816417 A CN202310816417 A CN 202310816417A CN 116860166 A CN116860166 A CN 116860166A
- Authority
- CN
- China
- Prior art keywords
- metadata
- storage device
- data
- migrated
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 80
- 238000013508 migration Methods 0.000 title claims abstract description 80
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000008569 process Effects 0.000 claims abstract description 24
- 230000008859 change Effects 0.000 claims description 36
- 238000012216 screening Methods 0.000 claims description 7
- 238000012217 deletion Methods 0.000 claims description 5
- 230000037430 deletion Effects 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000012545 processing Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 239000004973 liquid crystal related substance Substances 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data migration method, a data migration device, electronic equipment and a storage medium. The data migration method is applied to a process of migrating a data set to be migrated in a first storage device to a second storage device, wherein the data set to be migrated comprises a plurality of metadata and objects corresponding to the metadata, and the method comprises the following steps: in the second storage device, distributing corresponding metadata storage space and object storage space for target metadata in the data set to be migrated and an object corresponding to the target metadata; migrating an object corresponding to the target metadata to the object storage space; and under the condition that the object corresponding to the target metadata is migrated to the object storage space, updating the target metadata, and storing the updated target metadata to the metadata storage space. The technology of the application can improve the data migration efficiency of metadata and objects corresponding to the metadata.
Description
Technical Field
The present application relates to the field of data storage, and in particular, to a data migration method, apparatus, electronic device, and storage medium.
Background
Object Storage (Object Storage), also known as Object-based Storage, is a method used to describe solving and processing discrete units, i.e., objects (objects). Object storage has no directory hierarchy and no data format restrictions, can store any amount of data, and does not require partition management, and therefore object storage has been widely used. However, in the application process of object storage, after the object storage system is used for a long time, there is a problem that the storage device in the object storage system is old, so that metadata in the old storage device and an object corresponding to the metadata need to be migrated to other storage devices.
In the related art, when data migration is performed on metadata and an object corresponding to the metadata, the migration policy adopted is often: the method comprises the steps of calling an interface provided by S3 (Simple Storage Service ) from the outside of an object storage system to perform data migration on metadata and objects corresponding to the metadata, wherein the migration strategy is to scan and read the metadata and the objects corresponding to the metadata from old storage equipment, and then performing network transmission after S3 protocol conversion, so that the metadata and the objects corresponding to the metadata are written into other storage equipment.
The migration policy can implement data migration of metadata and objects corresponding to the metadata, but the data migration process is complex, and in the process of performing network transmission on metadata and objects corresponding to the metadata after the conversion of the S3 protocol, the migration policy is limited by the network bandwidth of the object storage system, so that the data migration efficiency of metadata and objects corresponding to the metadata is often low. Therefore, how to improve the data migration efficiency for metadata and objects corresponding to metadata is a technical problem to be solved.
In other scenarios, there may be a technical problem of how to improve data migration efficiency for metadata and objects corresponding to metadata.
Disclosure of Invention
The application provides a data migration method, a data migration device, electronic equipment and a storage medium, which are used for improving data migration efficiency of metadata and objects corresponding to the metadata.
According to an aspect of the present application, there is provided a data migration method applied to a process of migrating a data set to be migrated in a first storage device to a second storage device, where the data set to be migrated includes a plurality of metadata and objects corresponding to the metadata, and the method includes:
In the second storage device, aiming at target metadata in the data set to be migrated and an object corresponding to the target metadata, corresponding metadata storage space and object storage space are allocated;
migrating an object corresponding to the target metadata to an object storage space;
and under the condition that the object corresponding to the target metadata is migrated to the object storage space, updating the target metadata, and storing the updated target metadata to the metadata storage space.
According to a second aspect of the present application, there is provided a data migration apparatus applied to a process of migrating a data set to be migrated in a first storage device to a second storage device, where the data set to be migrated includes a plurality of metadata and objects corresponding to the metadata, the apparatus including:
the space allocation module is used for allocating corresponding metadata storage space and object storage space for target metadata and objects corresponding to the target metadata in the data set to be migrated in the second storage device;
the object migration module is used for migrating the object corresponding to the target metadata to the object storage space;
and the metadata updating module is used for updating the target metadata under the condition that the object corresponding to the target metadata is migrated to the object storage space, and storing the updated target metadata to the metadata storage space.
According to another aspect of the present application, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the embodiments of the present application.
According to another aspect of the application, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any of the embodiments of the application.
In the technology of the application, in the process of migrating the data set to be migrated in the first storage device to the second storage device, corresponding metadata storage space and object storage space are allocated for target metadata and objects corresponding to the target metadata in the data set to be migrated in the second storage device. And then, migrating the object corresponding to the target metadata to the object storage space, and updating the target metadata after migrating the object corresponding to the target metadata to the object storage space. And storing the updated target metadata into a metadata storage space.
After the corresponding metadata storage space and the object storage space are allocated for the target metadata and the objects corresponding to the target metadata, the objects corresponding to the target metadata can be directly migrated to the object storage space, and the updated target metadata can be stored in the metadata storage space. Therefore, the data migration process is simpler, and therefore the data migration efficiency of metadata and objects corresponding to the metadata can be improved.
In addition, since the interface provided by S3 is not required to be called from outside the object storage system, data migration is performed on metadata and an object to which the metadata corresponds. Therefore, in the process of migrating the data set to be migrated in the first storage device to the second storage device, the limitation of the network bandwidth of the object storage system is avoided. In this way, the data migration efficiency for the metadata and the object to which the metadata corresponds can be further improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the application or to delineate the scope of the application. Other features of the present application will become apparent from the description that follows.
Drawings
The drawings are included to provide a better understanding of the present application and are not to be construed as limiting the application. Wherein:
FIG. 1 is a flow chart of a data migration method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a data migration application process according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a data migration apparatus according to an embodiment of the present application;
fig. 4 is a schematic diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The present application provides a data migration method, and in particular, fig. 1 is a flowchart of a data migration method provided in an embodiment of the present application. The method is applied to a process of migrating a data set to be migrated in a first storage device to a second storage device, wherein the data set to be migrated comprises a plurality of metadata and objects corresponding to the metadata, and specifically comprises the following steps of:
Step S101: and in the second storage device, aiming at the target metadata in the data set to be migrated and the object corresponding to the target metadata, distributing corresponding metadata storage space and object storage space.
Step S102: and migrating the object corresponding to the target metadata to the object storage space.
Step S103: and under the condition that the object corresponding to the target metadata is migrated to the object storage space, updating the target metadata, and storing the updated target metadata to the metadata storage space.
In the data migration method provided by the embodiment of the application, in the process of migrating the data set to be migrated in the first storage device to the second storage device, the data set to be migrated is firstly stored in the second storage device. And allocating corresponding metadata storage space and object storage space for target metadata and objects corresponding to the target metadata in the data set to be migrated. And then, migrating the object corresponding to the target metadata to the object storage space, updating the target metadata after migrating the object corresponding to the target metadata to the object storage space, and storing the updated target metadata to the metadata storage space.
After the corresponding metadata storage space and the object storage space are allocated for the target metadata and the objects corresponding to the target metadata, the objects corresponding to the target metadata can be directly migrated to the object storage space, and the updated target metadata can be stored in the metadata storage space. Therefore, the data migration process is simpler, and therefore the data migration efficiency of metadata and objects corresponding to the metadata can be improved.
In addition, since the interface provided by S3 is not required to be called from outside the object storage system, data migration is performed on metadata and an object to which the metadata corresponds. Therefore, in the process of migrating the data set to be migrated in the first storage device to the second storage device, the limitation of the network bandwidth of the object storage system is avoided. In this way, the data migration efficiency for the metadata and the object to which the metadata corresponds can be further improved.
In the embodiment of the application, the object may include an image, audio, video, a log, or the like.
The first storage device and the second storage device are generally both storage devices for storing metadata and objects corresponding to the metadata. The first storage device refers to a storage device needing data migration, and the second storage device refers to metadata and an object corresponding to the metadata needs to be migrated. In an example, the first storage device generally refers to an older storage device in the object storage system for storing metadata, objects to which the metadata corresponds. An object storage system often refers to a system composed of a plurality of storage devices, a control unit, and devices (hardware) and algorithms (software) that manage information scheduling, and the like. So-called second storage devices tend to be new storage devices.
Since each time an object corresponding to target metadata is migrated from a first storage device to a second storage device and updated target metadata is stored in metadata is migrated from the first storage device to the second storage device, a data I/O (Input/Output) operation is performed. In order to reduce the number of I/O operations in the data migration process, so as to reduce the time and hardware resources required for data migration and improve the efficiency of data migration, when determining the target metadata and the object corresponding to the target metadata, part of metadata may be read from the data set to be migrated. And then screening a plurality of objects stored in the continuous physical addresses of the first storage device from the objects corresponding to the partial metadata according to the partial metadata. Finally, the plurality of objects stored in the consecutive physical addresses of the first storage device are determined as objects corresponding to the target metadata, and the metadata corresponding to the plurality of objects stored in the consecutive physical addresses of the first storage device are determined as target metadata.
The manner of reading part of metadata from the data set to be migrated may be: and in the process of migrating the data set to be migrated in the first storage device to the second storage device, scanning and reading part of metadata from the data set to be migrated in sequence.
The filtering out, according to the partial metadata, a plurality of objects stored in consecutive physical addresses of the first storage device from the objects corresponding to the partial metadata may refer to: after the partial metadata is acquired, determining the physical address of the object corresponding to the partial metadata and the object corresponding to the partial metadata in the first storage device according to the partial metadata. And then screening a plurality of objects stored in the continuous physical addresses of the first storage device from the objects corresponding to the partial metadata according to the physical addresses of the objects corresponding to the partial metadata in the first storage device.
By determining a plurality of objects stored in consecutive physical addresses of the first storage device as objects to which target metadata corresponds, and determining metadata corresponding to a plurality of objects stored in consecutive physical addresses of the first storage device as target metadata may mean: and screening out a plurality of objects stored in the continuous physical addresses of the first storage device from the objects corresponding to the partial metadata as objects corresponding to the target metadata, and screening out metadata corresponding to the plurality of objects stored in the continuous physical addresses of the first storage device from the partial metadata as the target metadata.
That is, in the embodiment of the present application, a plurality of objects stored in consecutive physical addresses of a first storage device are migrated from the first storage device to a second storage device at one time, metadata corresponding to the plurality of objects stored in consecutive physical addresses of the first storage device are updated, and the updated metadata is stored in the second storage device. Therefore, the number of I/O operations in the data migration process can be reduced, the time and hardware resources consumed by the data migration are reduced, and the data migration efficiency is improved.
After the object corresponding to the target metadata is migrated from the first storage device to the second storage device, the data attribute (e.g., storage time of the object, storage space of the object, and data format of the object) of the object corresponding to the target metadata may be changed. That is, after the object corresponding to the target metadata is migrated from the first storage device to the second storage device, metadata corresponding to the migrated object may change from metadata corresponding to the original metadata. Therefore, after the object corresponding to the target metadata is migrated from the first storage device to the second storage device, the target metadata needs to be updated according to the change condition of the data attribute of the object corresponding to the target metadata. Then, the updated target metadata is stored in the metadata storage space.
In the process of migrating the data set to be migrated in the first storage device to the second storage device, in order to prevent the data access request for the first storage device from being affected by data migration, the data migration method provided by the embodiment of the application judges whether the data access request is a data reading request or not after the data access request for the first storage device is acquired. Then, when the data access request is a data read request, the data access request is allocated to the first storage device, or when the object for which the data access request is directed has been migrated to the second storage device, the data access request is randomly allocated to the first storage device or the second storage device. That is, when the data access request is a data read request, the storage device to which the data access request is directed is determined to be the first storage device, or the storage device to which the data access request is directed is randomly determined to be the first storage device or the second storage device.
In addition, if the data access request is not a data read request, the data access operation is executed according to the request type corresponding to the data access request.
The determination of the storage device to which the data access request is directed as the first storage device means that the computing device that sent the data access request is instructed to access the first storage device, and the determination of the storage device to which the data access request is directed as the second storage device means that the computing device that sent the data access request is instructed to access the second storage device.
When the data access request is a data reading request, the data access request is distributed to the first storage device, or when the object aimed by the data access request is migrated to the second storage device, the data access request is randomly distributed to the first storage device or the second storage device, so that in the process of migrating the data set to be migrated in the first storage device to the second storage device, the object and the corresponding metadata in the data set to be migrated can still be read normally.
The request type includes a data write request, a data delete request, a data update request, and the like. When the data access request is a data writing request aiming at the first object, the first object and metadata corresponding to the first object are written into the second storage device, and then a first data change record is recorded in a preset database. The first data change record is at least used for indicating that the first object is written to the second storage device.
Specifically, when the data access request is a data write-in request for the first object, the storage device for which the data access request is directed is determined to be the second storage device, so as to instruct the computing device sending the data access request to access the second storage device, and thereby control the computing device to write the first object and metadata corresponding to the first object into the second storage device. And then, recording the first data change record in a preset database.
When the data access request is a data write request for the first object, the storage device for which the data access request is made is determined to be the second storage device, and the data write request originally for the first storage device can be converted into a request for the second storage device. Therefore, the method and the device can ensure that the data writing request can be normally responded, simultaneously, the computing device sending the data access request is directly instructed to access the second storage device, and the first object and the corresponding metadata can be directly written into the second storage device, so that the need of writing the first object and the corresponding metadata into the first storage device can be avoided. And then, the first object and the corresponding metadata are migrated to the second storage device, so that the data migration efficiency is improved, and the processing resources required to be consumed by the data migration are reduced.
When the data access request is a data deletion request for the second object, it is then necessary to first determine whether the second object has been migrated to the second storage device. And then deleting the second object and the metadata corresponding to the second object in the second storage device when the second object is migrated to the second storage device. And then, recording a second data change record in a preset database, and marking the second object as the deleted object in the first storage device. The second object is an object in the data set to be migrated, and the second data change record is at least used for indicating that the second object has been deleted.
In addition, when the second object is not migrated to the second storage device, the second data change record is recorded in a preset database, and the second object is marked as the deleted object in the first storage device. Wherein the second data change record is at least used to indicate that the second object has been deleted. When a second object is marked as a deleted object, no data migration is performed on the second object.
When the data access request is a data deletion request for the second object, deleting the second object and metadata corresponding to the second object in the second storage device directly if the second object is migrated to the second storage device, and marking the second object as the deleted object in the first storage device if the second object is not migrated to the second storage device. Thus, not only the data deleting request for the second object can be responded normally, but also the deleted object can be prevented from being subjected to data migration, so that the efficiency of data migration can be improved, and the processing resources required to be consumed by data migration can be reduced.
When the data access request is a data update request for the third object, then it is necessary to first determine whether the third object has been migrated to the second storage device. And then, when the third object is migrated to the second storage device, reading the third object from the second storage device, and updating the data of the third object to obtain the updated third object. And meanwhile, updating the metadata corresponding to the third object to obtain the updated metadata corresponding to the third object. And writing the updated third object and metadata corresponding to the updated third object into the second storage device, recording a third data change record in a preset database, and marking the third object as an updated object in the first storage device. The third object is an object in the data set to be migrated, and the third data change record is at least used for indicating that the third object is updated to be the updated third object.
In addition, if the third object is not migrated to the second storage device, the third object is read from the first storage device, and data update is performed on the third object to obtain an updated third object. And meanwhile, updating the metadata corresponding to the third object to obtain the updated metadata corresponding to the third object. And writing the updated third object and metadata corresponding to the updated third object into the second storage device, recording a third data change record in a preset database, and marking the third object as an updated object in the first storage device.
When the data access request is a data update request for the third object, after the updated third object and metadata corresponding to the updated third object are obtained, the updated third object and the metadata corresponding to the updated third object are written into the second storage device, and the third object is marked as an updated object in the first storage device. Thus, not only the data update request for the third object can be normally responded, but also the data migration of the updated object can be avoided, so that the efficiency of the data migration can be improved, and the processing resources required to be consumed by the data migration can be reduced. In addition, the updated third object and the metadata corresponding to the updated third object are directly written into the second storage device, so that the efficiency of data migration can be further improved.
It should be noted that, the execution body of the data migration method provided in the embodiment of the present application may be a data migration system configured in a set computing device. Fig. 2 is a schematic diagram of a data migration application process according to an embodiment of the application. Included in fig. 2 is a data migration system 201 that includes a temporary takeover system 201-1, a data change record database 201-2, a migration scheduling system 201-3, a first storage device 202, a second storage device 203, and a forwarding adapter 203-1.
In practical applications, the second storage device 203 may be pointed to the temporary takeover system 201-1 through the forwarding adapter 203-1 configured with haproxy (free and open source code software), so that the temporary takeover system 201-1 has forwarding capability and takes over the data set to be migrated, thereby making the data set to be migrated a static aggregate. After the temporary takeover system 201-1 takes over the set of data to be migrated, virtual IP (Internet Protocol, internetworking protocol) originally directed to the first storage device 202 may be directed to the second storage device 203. In this case, the computing device may acquire the data access request sent by the first storage device 202 by the temporary takeover system 201-1, and when the temporary takeover system 201-1 acquires the data access request, it may first determine whether the data access request is a data read request, and then, when the data access request is a data read request, allocate the data access request to the first storage device 202, or when the object for which the data access request is directed has been migrated to the second storage device 203, randomly allocate the data access request to the first storage device 202 or the second storage device 203. If the data access request is not a data read request, the temporary takeover system 201-1 performs a data access operation according to the request type corresponding to the data access request. Specifically, when the data access request is a data writing request for the first object, the storage device for which the data access request is made is determined as the second storage device 203, so as to instruct the computing device sending the data access request to access the second storage device 203, thereby controlling the computing device to write the first object and metadata corresponding to the first object into the second storage device 203, and then recording the first data change record in the preset data change record database 201-2.
In addition, when the data access request is a data deletion request for the second object, it is required to determine whether the second object has been migrated to the second storage device 203, and then delete the second object and metadata corresponding to the second object in the second storage device 203 when the second object has been migrated to the second storage device 203. Thereafter, a second data change record is recorded in the preset data change record database 201-2, and the second object is marked as an object that has been deleted in the first storage device 202. And when the second object is not migrated to the second storage device 203, the second data change record is recorded in the preset data change record database 201-2, and the second object is marked as the deleted object in the first storage device 202.
In addition, when the data access request is a data update request for the third object, it is required to determine whether the third object has been migrated to the second storage device 203, then, when the third object has been migrated to the second storage device 203, read the third object from the second storage device 203, and perform data update on the third object to obtain an updated third object, and update metadata corresponding to the third object to obtain metadata corresponding to the updated third object, then, write the updated third object and metadata corresponding to the updated third object into the second storage device 203, record a third data change record in the preset data change record database 201-2, and mark the third object as an updated object in the first storage device 202. If the third object is not migrated to the second storage device 203, the third object is read from the first storage device 202, and the data of the third object is updated to obtain an updated third object, and at the same time, the metadata corresponding to the third object is updated to obtain metadata corresponding to the updated third object, and then the updated third object and the metadata corresponding to the updated third object are written into the second storage device 203, and a third data change record is recorded in the preset data change record database 201-2, and the third object is marked as an updated object in the first storage device 202.
In migrating the data set to be migrated in the first storage device 202 to the second storage device 203, the migration scheduling system 201-3 may scan and read part of metadata from the data set to be migrated in sequence, then filter out a plurality of objects stored in continuous physical addresses of the first storage device 202 from objects corresponding to the part of metadata, and then determine the plurality of objects stored in the continuous physical addresses of the first storage device 202 as objects corresponding to target metadata, and determine metadata corresponding to the plurality of objects stored in the continuous physical addresses of the first storage device 202 as target metadata. After determining the target metadata and the objects corresponding to the target metadata, the migration scheduling system 201-3 reads the available storage space in the second storage device 203, allocates the corresponding metadata storage space and the object storage space for the target metadata and the objects corresponding to the target metadata in the data set to be migrated in the available storage space, migrates the objects corresponding to the target metadata to the object storage space, and further updates the target metadata and stores the updated target metadata to the metadata storage space when the objects corresponding to the target metadata migrate to the object storage space. Each time the object corresponding to the target metadata is migrated to the object storage space and the updated target metadata is stored in the metadata storage space, the migration scheduling system 201-3 writes a data migration record into the data change record database 201-2, where the data migration record is used to represent the object that has been migrated to the second storage device and the corresponding metadata.
When the first storage device 202 and the second storage device 203 are two storage devices corresponding to different data types, the metadata adapter and the object adapter may be configured at the first storage device 202 and the second storage device 203, respectively, to enable the migration scheduling system 201-3 to migrate the object stored in the first storage device 202 and the corresponding metadata to the second storage device 203 in the first data type and store in the second storage device 203 in the second data type. Wherein the first data type and the second data type belong to different data types.
After migrating the set of data to be migrated in the first storage device to the second storage device, forwarding adapter 203-1 may be removed such that second storage device 203 is no longer pointed to temporary takeover system 201-1, thereby enabling the second storage device to replace the first storage device.
Corresponding to the data migration method provided by the embodiment of the application, the embodiment of the application also provides a data migration device. Fig. 3 is a block diagram of a data migration apparatus 300 according to an embodiment of the present application, where the apparatus 300 is applied to a process of migrating a data set to be migrated in a first storage device to a second storage device, the data set to be migrated includes a plurality of metadata and objects corresponding to the metadata, and the apparatus may include:
The space allocation module 301 is configured to allocate, in the second storage device, a corresponding metadata storage space and an object storage space for target metadata and an object corresponding to the target metadata in the data set to be migrated;
the object migration module 302 is configured to migrate an object corresponding to the target metadata to the object storage space;
and the metadata updating module 303 is configured to update the target metadata when the object corresponding to the target metadata is migrated to the object storage space, and store the updated target metadata in the metadata storage space.
In one possible implementation, the space allocation module 301 includes:
the metadata reading sub-module is used for reading partial metadata from the data set to be migrated;
the object screening sub-module is used for screening a plurality of objects stored in the continuous physical addresses of the first storage device from the objects corresponding to the partial metadata according to the partial metadata;
and the data determination submodule is used for determining a plurality of objects stored in the continuous physical address of the first storage device as objects corresponding to target metadata and determining metadata corresponding to the plurality of objects stored in the continuous physical address of the first storage device as the target metadata.
In one possible implementation, the apparatus further includes:
a data access request acquisition unit configured to acquire a data access request for a first storage device;
a data access request judging unit for judging whether the data access request is a data reading request;
the data access request allocation unit is used for allocating the data access request to the first storage device when the data access request is a data reading request or randomly allocating the data access request to the first storage device or the second storage device when an object aimed by the data access request is migrated to the second storage device.
In a possible implementation manner, the data access request allocation unit is further configured to perform the data access operation according to a request type corresponding to the data access request when the data access request is not the data read request.
In one possible implementation manner, the data access request allocation unit is specifically configured to, when the data access request is a data write request for the first object, write the first object and metadata corresponding to the first object to the second storage device; recording a first data change record in a preset database; the first data change record is used to indicate at least that the first object is written to the second storage device.
In one possible implementation, the request type includes a data delete request; the data access request allocation unit is specifically configured to determine, when the data access request is a data deletion request for the second object, whether the second object has been migrated to the second storage device; the second object is an object in the data set to be migrated; if so, deleting the second object and the metadata corresponding to the second object in the second storage device; recording a second data change record in a preset database, and marking the second object as a deleted object in the first storage device; the second data change record is at least used to indicate that the second object has been deleted.
In one possible implementation manner, the data access request allocation unit is further configured to record a second data change record in a preset database when the second object is not migrated to the second storage device, and mark the second object as a deleted object in the first storage device; the second data change record is at least used to indicate that the second object has been deleted.
The functions of each module in each device of the embodiment of the present application may be referred to the corresponding descriptions in the above methods, and have corresponding beneficial effects, which are not described herein.
The application further provides the electronic equipment and the readable storage medium.
Fig. 4 shows a schematic block diagram of an example electronic device 400 that may be used to implement an embodiment of the application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
As shown in fig. 4, the apparatus 400 includes a computing unit 401 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 402 or a computer program loaded from a storage unit 408 into a Random Access Memory (RAM) 403. In RAM 403, various programs and data required for the operation of device 400 may also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other by a bus 404. An input/output (I/O) interface 406 is also connected to bus 404.
The various components in device 400 are connected to I/O interface 406, including: an input unit 406 such as a keyboard, a mouse, etc.; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 401 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the data migration method. For example, in some embodiments, the data migration method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 400 via the ROM 402 and/or the communication unit 409. When a computer program is loaded into RAM 403 and executed by computing unit 401, one or more steps of the data migration method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the data migration method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit a network of data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present application may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, so long as the desired result of the technical solution of the present application is achieved, and the present application is not limited herein.
The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.
Claims (10)
1. The data migration method is characterized by being applied to a process of migrating a data set to be migrated in a first storage device to a second storage device, wherein the data set to be migrated comprises a plurality of metadata and objects corresponding to the metadata, and the method comprises the following steps:
in the second storage device, distributing corresponding metadata storage space and object storage space for target metadata in the data set to be migrated and an object corresponding to the target metadata;
Migrating an object corresponding to the target metadata to the object storage space;
and under the condition that the object corresponding to the target metadata is migrated to the object storage space, updating the target metadata, and storing the updated target metadata to the metadata storage space.
2. The method of claim 1, wherein the target metadata and the object to which the target metadata corresponds are determined by:
reading partial metadata from the data set to be migrated;
screening a plurality of objects stored in continuous physical addresses of the first storage device from the objects corresponding to the partial metadata according to the partial metadata;
and determining a plurality of objects stored in the continuous physical address of the first storage device as objects corresponding to the target metadata, and determining metadata corresponding to the plurality of objects stored in the continuous physical address of the first storage device as the target metadata.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
acquiring a data access request for the first storage device;
Judging whether the data access request is a data reading request or not;
if yes, the data access request is distributed to the first storage device, or when the object aimed by the data access request is migrated to the second storage device, the data access request is randomly distributed to the first storage device or the second storage device.
4. A method according to claim 3, characterized in that the method further comprises:
and if the data access request is not the data reading request, executing data access operation according to the request type corresponding to the data access request.
5. The method of claim 4, wherein the request type comprises a data write request; the executing the data access operation according to the request type corresponding to the data access request includes:
when the data access request is a data write-in request aiming at a first object, writing the first object and metadata corresponding to the first object into the second storage device;
recording a first data change record in a preset database; the first data change record is at least used to indicate that the first object is written to the second storage device.
6. The method of claim 4, wherein the request type comprises a data delete request; the executing the data access operation according to the request type corresponding to the data access request includes:
determining whether a second object has been migrated to the second storage device when the data access request is a data deletion request for the second object; the second object is an object in the data set to be migrated;
if yes, deleting the second object and metadata corresponding to the second object in the second storage device;
recording a second data change record in a preset database, and marking the second object as a deleted object in the first storage device; the second data change record is at least used to indicate that the second object has been deleted.
7. The method of claim 6, wherein the method further comprises:
when the second object is not migrated to the second storage device, recording a second data change record in a preset database, and marking the second object as a deleted object in the first storage device; the second data change record is at least used to indicate that the second object has been deleted.
8. The data migration apparatus is characterized in that the data migration apparatus is applied to a process of migrating a data set to be migrated in a first storage device to a second storage device, wherein the data set to be migrated includes a plurality of metadata and objects corresponding to the metadata, and the apparatus includes:
the space allocation module is used for allocating corresponding metadata storage space and object storage space for target metadata in the data set to be migrated and objects corresponding to the target metadata in the second storage device;
the object migration module is used for migrating the object corresponding to the target metadata to the object storage space;
and the metadata updating module is used for updating the target metadata under the condition that the object corresponding to the target metadata is migrated to the object storage space, and storing the updated target metadata to the metadata storage space.
9. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; characterized in that the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310816417.4A CN116860166A (en) | 2023-07-04 | 2023-07-04 | Data migration method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310816417.4A CN116860166A (en) | 2023-07-04 | 2023-07-04 | Data migration method, device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116860166A true CN116860166A (en) | 2023-10-10 |
Family
ID=88226258
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310816417.4A Pending CN116860166A (en) | 2023-07-04 | 2023-07-04 | Data migration method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116860166A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108021336A (en) * | 2016-10-31 | 2018-05-11 | 甲骨文国际公司 | Data layout scheme for seamless Data Migration |
| JP2019152945A (en) * | 2018-03-01 | 2019-09-12 | Ｎｅｃソリューションイノベータ株式会社 | Storage device, data migration method and program |
| CN110427350A (en) * | 2019-07-11 | 2019-11-08 | 北京梧桐车联科技有限责任公司 | Data processing method and device, storage equipment and medium in Data Migration |
| CN113296696A (en) * | 2021-03-02 | 2021-08-24 | 阿里巴巴新加坡控股有限公司 | Data access method, computing device and storage medium |
| CN114064563A (en) * | 2020-07-30 | 2022-02-18 | 深圳市杉岩数据技术有限公司 | Data migration method and server based on object storage |
| CN114489472A (en) * | 2020-10-26 | 2022-05-13 | 三星电子株式会社 | Method of operating storage device and method of operating storage system using the same |
-
2023
- 2023-07-04 CN CN202310816417.4A patent/CN116860166A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108021336A (en) * | 2016-10-31 | 2018-05-11 | 甲骨文国际公司 | Data layout scheme for seamless Data Migration |
| JP2019152945A (en) * | 2018-03-01 | 2019-09-12 | Ｎｅｃソリューションイノベータ株式会社 | Storage device, data migration method and program |
| CN110427350A (en) * | 2019-07-11 | 2019-11-08 | 北京梧桐车联科技有限责任公司 | Data processing method and device, storage equipment and medium in Data Migration |
| CN114064563A (en) * | 2020-07-30 | 2022-02-18 | 深圳市杉岩数据技术有限公司 | Data migration method and server based on object storage |
| CN114489472A (en) * | 2020-10-26 | 2022-05-13 | 三星电子株式会社 | Method of operating storage device and method of operating storage system using the same |
| CN113296696A (en) * | 2021-03-02 | 2021-08-24 | 阿里巴巴新加坡控股有限公司 | Data access method, computing device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 周延森: "《操作系统原理》", 北京邮电大学出版社, pages: 171 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2017107414A1 (en) | File operation method and device | |
| US11150949B2 (en) | Resource release method, resource allocation method, devices, and computer program products | |
| CN113641457A (en) | Container creation method, device, apparatus, medium, and program product | |
| EP4170497A1 (en) | Access control method and apparatus for shared memory, electronic device and autonomous vehicle | |
| CN110520844A (en) | Cloud management platform, virtual machine management method and its system | |
| CN113806300B (en) | Data storage method, system, device, equipment and storage medium | |
| WO2023066182A1 (en) | File processing method and apparatus, device, and storage medium | |
| US11237761B2 (en) | Management of multiple physical function nonvolatile memory devices | |
| WO2018000300A1 (en) | Data operation method for electronic device, and electronic device | |
| CN110990356B (en) | Real-time automatic capacity expansion method and system for logical mirror image | |
| CN113535087B (en) | Data processing method, server and storage system in data migration process | |
| US11403026B2 (en) | Method, device and computer program product for managing storage system | |
| CN113127438B (en) | Method, apparatus, server and medium for storing data | |
| CN108205559B (en) | Data management method and equipment thereof | |
| CN116860166A (en) | Data migration method, device, electronic equipment and storage medium | |
| US11977559B2 (en) | Providing instant and distributed access to a source blob via copy-on-read blobs and link blobs | |
| US10678453B2 (en) | Method and device for checking false sharing in data block deletion using a mapping pointer and weight bits | |
| US11385822B2 (en) | Moving data from a first group of slices to a second group of slices | |
| CN113127430B (en) | Mirror image information processing method, mirror image information processing device, computer readable medium and electronic equipment | |
| CN113051244A (en) | Data access method and device, and data acquisition method and device | |
| CN110058790B (en) | Method, apparatus and computer program product for storing data | |
| CN110674090A (en) | Resource file processing method and device | |
| CN113986134B (en) | Method for storing data, method and device for reading data | |
| CN114615273B (en) | Data transmission method, device and equipment based on load balancing system | |
| US20220353874A1 (en) | Method and Apparatus for Maximizing a Number of Connections That Can Be Executed from a Mobile Application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |