CN113836116A - Data migration method and device, electronic equipment and readable storage medium - Google Patents
Data migration method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN113836116A CN113836116A CN202111149417.0A CN202111149417A CN113836116A CN 113836116 A CN113836116 A CN 113836116A CN 202111149417 A CN202111149417 A CN 202111149417A CN 113836116 A CN113836116 A CN 113836116A
- Authority
- CN
- China
- Prior art keywords
- data
- storage pool
- subdata
- storage
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 129
- 230000005012 migration Effects 0.000 title claims abstract description 128
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000004590 computer program Methods 0.000 claims description 11
- 230000011218 segmentation Effects 0.000 claims description 2
- 230000004044 response Effects 0.000 abstract description 7
- 238000012545 processing Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 230000004048 modification Effects 0.000 description 8
- 238000012986 modification Methods 0.000 description 8
- 230000002159 abnormal effect Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 6
- 238000012217 deletion Methods 0.000 description 6
- 230000037430 deletion Effects 0.000 description 6
- 238000004140 cleaning Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data migration method, a data migration device, electronic equipment and a readable storage medium. The method comprises the steps of constructing a first storage pool and a second storage pool which have different storage performances in advance, wherein the storage performance of the first storage pool is superior to that of the second storage pool; dividing target data into first subdata and second subdata, storing the first subdata to a first storage pool, and storing the second subdata to a second storage pool; when a migration instruction of target data is received, writing the first subdata read from the first storage pool into the second storage pool, and simultaneously sending a request for updating storage position information of the target data to a metadata server; when the metadata server responses to the completion of updating, the first subdata is deleted from the first storage pool to complete the migration operation of the target data, so that the error probability of file migration can be reduced on the basis of improving the file migration efficiency.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data migration method and apparatus, an electronic device, and a readable storage medium.
Background
With the rapid development of computer technology and the wide application of computer data in daily life, various computer data are increasing explosively, and correspondingly, storage technology is also developing correspondingly. Distributed file system applications arise in order to accommodate the fast, secure storage of data. The distributed file system is a cluster formed by a plurality of file storage node servers, files are stored in blocks, an object is used as a basic unit to support one part of data to be stored on a plurality of nodes, each node can acquire complete data through communication among the nodes, when the nodes are down, the complete data can be recovered according to a configured strategy, and the distributed file system has the advantages of being high in availability, performance and expansibility and the like, wherein each node provides a metadata service (MDS) which is used for various access operations of the metadata and balancing business pressure.
For a distributed file system, file migration, which may also be referred to as file layering, is inevitable, and the related art performs the file migration or file layering process as follows: the migration client reads all data of the file from the high-level pool; the migration client writes data into the slow pool; updating metadata to a metadata server MDS, namely modifying a slow pool of a data pool of the file from a high pool, and updating the response client by the MDS; after receiving the response, the client deletes the data of the advanced pool; migration success/failure.
However, such file layering is only suitable for a scenario where file data is stored in a single data pool, when a file is stored in a plurality of data pools, data to be read, written and deleted are different during file layering migration, and at the same time, exceptions, i.e., read errors, write errors, metadata modification errors, deletion errors, and the like may occur at each step.
In view of this, how to improve the file migration efficiency and reduce the error probability of file migration is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
The application provides a data migration method, a data migration device, an electronic device and a readable storage medium, which can improve file migration efficiency and reduce file migration error probability.
In order to solve the above technical problems, embodiments of the present invention provide the following technical solutions:
an embodiment of the present invention provides a data migration method, including:
pre-constructing a first storage pool and a second storage pool which have different storage performances, wherein the storage performance of the first storage pool is superior to that of the second storage pool;
dividing target data into first subdata and second subdata; the first subdata is stored in the first storage pool, and the second subdata is stored in the second storage pool;
when a migration instruction of the target data is received, writing the first subdata read from the first storage pool into the second storage pool, and simultaneously sending a request for updating storage position information of the target data to a metadata server;
and when the metadata server responds to the completion of updating, deleting the first subdata from the first storage pool so as to complete the migration operation of the target data.
Optionally, the writing the first sub data read from the first storage pool to the second storage pool includes:
reading the first subdata from the first storage pool;
judging whether the first subdata is successfully read from the first storage pool or not;
if the first subdata is successfully read from the first storage pool, writing the first subdata into the second storage pool;
and if the first subdata is not successfully read from the first storage pool, deleting the junk data in the second storage pool and simultaneously sending migration failure information.
Optionally, after writing the first sub data read from the first storage pool to the second storage pool, the method includes:
judging whether the first subdata is successfully written into the second storage pool or not;
and if the first subdata is not successfully written into the second storage pool, deleting the junk data in the second storage pool, and simultaneously sending migration failure information.
Optionally, after sending the request for updating the storage location information of the target data to the metadata server, the method further includes:
judging whether the metadata server successfully updates the metadata;
if the metadata server does not successfully update the metadata, deleting the junk data in the second storage pool, and sending migration failure information;
and if the metadata server successfully updates the metadata, deleting the junk data in the first storage pool.
Optionally, after deleting the garbage data in the first storage pool, the method further includes:
if the data deleting operation in the first storage pool is successfully completed, the data migration is successful;
and if the data deleting operation in the first storage pool is not completed successfully, the data migration fails, and the junk data in the first storage pool is continuously deleted when the data migration operation is executed next time.
Optionally, the dividing the target data into the first sub-data and the second sub-data includes:
when target data to be written is received, judging whether the target data is larger than a preset capacity threshold value;
if the target data is larger than a preset capacity threshold, dividing the target data into first subdata and second subdata; the space occupation capacity value of the first subdata is smaller than that of the second subdata;
and if the target data is less than or equal to a preset capacity threshold value, storing the target data to the first storage pool.
Optionally, the sending the request for updating the storage location information of the target data to the metadata server includes:
and sending an instruction for updating the data pool into the second storage pool to the metadata server.
Another aspect of the embodiments of the present invention provides a data migration apparatus, including:
the storage pool constructing module is used for constructing a first storage pool and a second storage pool with different storage performances in advance, wherein the storage performance of the first storage pool is superior to that of the second storage pool;
the data segmentation storage module is used for segmenting target data into first subdata and second subdata; the first subdata is stored in the first storage pool, and the second subdata is stored in the second storage pool;
a data migration module, configured to, when receiving a migration instruction of the target data, write the first sub-data read from the first storage pool into the second storage pool, and send a request for updating storage location information of the target data to a metadata server;
and the data deleting module is used for deleting the first subdata from the first storage pool when the metadata server responds that the updating is completed so as to complete the migration operation of the target data.
An embodiment of the present invention further provides an electronic device, which includes a processor, and the processor is configured to implement the steps of the data migration method according to any one of the foregoing items when executing the computer program stored in the memory.
Finally, an embodiment of the present invention provides a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the data migration method according to any one of the foregoing items.
The technical scheme provided by the application has the advantages that the target data to be written are divided into two parts based on file multi-data-pool storage, the two parts are respectively stored in the first storage pool and the second storage pool with different storage performances, when the files are migrated in a layered mode, only the data in the first storage pool with better performance is read and written into the second storage pool, the metadata is used for modifying 2 storage pools into 1 storage pool, and only the data in the first storage pool is deleted when the files are deleted, so that the data of layered reading, writing, metadata modification and old data deletion operation only has the data in the first storage pool, but not all the data of the files, the file migration efficiency can be effectively improved, the file migration error probability is reduced, file layered migration scenes are further enriched, the stability of a storage system is improved, and the competitiveness of products is improved.
In addition, the embodiment of the invention also provides a corresponding implementation device, electronic equipment and a readable storage medium for the data migration method, so that the method has higher practicability, and the device, the electronic equipment and the readable storage medium have corresponding advantages.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the related art, the drawings required to be used in the description of the embodiments or the related art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of another data migration method according to an embodiment of the present invention;
FIG. 3 is a block diagram of a data migration apparatus according to an embodiment of the present invention;
fig. 4 is a block diagram of an embodiment of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "comprising" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may include other steps or elements not expressly listed.
Having described the technical solutions of the embodiments of the present invention, various non-limiting embodiments of the present application are described in detail below.
Referring to fig. 1, fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present invention, where the embodiment of the present invention may include the following:
s101: a first storage pool and a second storage pool having different storage performances are pre-constructed.
The storage performance of the first storage pool of this embodiment is better than that of the second storage pool, and the first storage pool may be, for example, a high-level pool, that is, a data pool composed of a solid state disk SSD; the second storage pool may be a slow pool, i.e., a data pool comprised of mechanical hard disk HDDs. In this embodiment, two data pools with different performances are constructed for implementing storage of multiple file data pools, that is, data of a file to be written is stored in the 2 data pools, for example, the first 4M of the file to be written may be stored in a high-level pool, and the rest may be stored in a slow-rate pool.
S102: the target data is divided into first sub-data and second sub-data.
Because the size of a file cannot be identified when the file is created, the large file is written into a first storage pool such as a high-level pool, which wastes SSD high-cost storage medium resources, in order to take both storage performance and storage cost into consideration, the target data may be divided into two parts, the target data is stored into two storage pools, the first sub-data, such as the first half, is stored into the first storage pool, and the second sub-data, that is, the remaining data, is stored into the second storage pool.
S103: when a migration instruction of the target data is received, the first sub data read from the first storage pool is written into the second storage pool, and simultaneously a request for updating the storage position information of the target data is sent to the metadata server.
The migration instruction of this embodiment refers to file layering, that is, data is transferred between a plurality of storage media without being perceived by a user, that is, data is migrated. The data migration can support various strategies of users, and for data meeting the migration requirements, the system carries out the non-perception file data migration processing under the condition of not influencing normal services, so that the requirements of users on storage schemes of performance layering and data cold and hot layering can be met. The metadata server of this embodiment is used to cache metadata of the distributed file system and provide various access operations of the metadata, and the metadata server of this embodiment is also used to load a hierarchical index table and migrate a file. In the embodiment, after the file is migrated, the data storage location information in the metadata server is changed from 2 storage pools to 1 storage pool, and in order to facilitate metadata change, an instruction for updating the data pool to the second storage pool may be directly sent to the metadata server, so as to implement a metadata update operation.
S104: and when the metadata server responds to the completion of the updating, deleting the first subdata from the first storage pool so as to complete the migration operation of the target data.
When a file is migrated hierarchically in a large file, firstly, whether the file is stored in multiple data pools is determined, if so, only the front 4M data of the file in the first storage pool is migrated, and the rest is stored in the second storage pool, if so, only the front 4M data of the file in the first storage pool is migrated, the data in the whole process only has the 4M data, and when each migration stage is abnormal, only the 4M data is cleaned, namely, the data in the first storage pool before metadata is modified is valid, and the second storage pool after the metadata is modified is valid data. Therefore, after the data migration is completed, i.e., the metadata successfully modifies the data, the data in the first storage pool may be cleaned up.
In the technical scheme provided by the embodiment of the invention, target data to be written is divided into two parts based on file multi-data-pool storage, the two parts are respectively stored in a first storage pool and a second storage pool with different storage performances, when a file is hierarchically migrated, data in the first storage pool with better performance is read only and written into the second storage pool, metadata is modified into 1 storage pool from 2 storage pools, and only the data in the first storage pool is deleted when the file is deleted, so that the data of operations of reading, writing, metadata modification and data deletion in hierarchical mode only has the data in the first storage pool, but not all the data of the file, and therefore, the file migration efficiency can be effectively improved, the file migration error probability is reduced, file hierarchical migration scenes are further enriched, the stability of a storage system is improved, and the competitiveness of a product is improved.
In the above embodiment, if the file is recorded in 2 data pools, the migration client only reads corresponding first sub-data from the first storage pool, and then writes the data into the second storage pool, and does not read or write all data of the file any more, after the first sub-data is written into the second storage pool, the first sub-data informs that the metadata of the updated data pool is the second storage pool, at this time, the 2 data pools are changed to 1 data pool, the data is in the second storage pool, after the MDS finishes updating the metadata, the migration client is answered, and at this time, the migration client only deletes the first sub-data from the first storage pool; in the migration process, when reading fails, cleaning the garbage data in the slow pool, and failing in migration; when the writing fails, cleaning the garbage data in the slow pool, and failing to migrate; when updating metadata failure, clear up the junk data in the slow speed pond, when deleting failure, the migration fails, and the next migration clears up the junk data in the senior pond again to this further richens the file layering scene, promotes system stability, specifically can include:
as an alternative embodiment, the process of writing the first sub data read from the first storage pool to the second storage pool may include:
reading first subdata from a first storage pool; judging whether the first subdata is successfully read from the first storage pool or not; if the first subdata is successfully read from the first storage pool, writing the first subdata into the second storage pool; and if the first subdata is not successfully read from the first storage pool, deleting the junk data in the second storage pool and simultaneously sending migration failure information.
As another optional implementation, after writing the first sub data read from the first storage pool to the second storage pool, the method may further include:
judging whether the first subdata is successfully written into the second storage pool or not; and if the first subdata is not successfully written into the second storage pool, deleting the junk data in the second storage pool, and simultaneously sending migration failure information.
As another optional implementation, after sending the request for updating the storage location information of the target data to the metadata server, the method may further include:
judging whether the metadata server successfully updates the metadata; if the metadata server does not successfully update the metadata, deleting the junk data in the second storage pool, and sending migration failure information; and if the metadata server successfully updates the metadata, deleting the junk data in the first storage pool.
If the data deleting operation in the first storage pool is successfully completed, the data migration is successful;
and if the data deleting operation in the first storage pool is not completed successfully, the data migration fails, and the junk data in the first storage pool is continuously deleted when the data migration operation is executed next time.
In this embodiment, in order to improve the whole file migration efficiency, the space occupation capacity value of the first sub-data is smaller than that of the second sub-data; for example, the first sub data is the first 4M data of the target data. When a file is migrated, reading the front 4M data of the file only from the first storage pool, writing the data into the second storage pool, changing the data pool into the second pool when the metadata is updated, then deleting the front 4M data of the file only in the first storage pool, aiming at the abnormality of each stage, the front 4M data of the file in the second storage pool is cleared, the part of data is junk data, if the front 4M data of the file in the first storage pool is the second storage pool after the metadata is changed, the front 4M data of the file in the first storage pool is cleared, and the part of data is junk data, so that the file migration efficiency is effectively improved, and the error probability in the file migration process is reduced.
Based on the foregoing embodiment, in order to improve the flexibility of the whole file migration and enhance the user experience, based on the foregoing embodiment, the process of dividing the target data into the first sub-data and the second sub-data may include:
when target data to be written is received, judging whether the target data is larger than a preset capacity threshold value;
if the target data is larger than the preset capacity threshold, dividing the target data into first subdata and second subdata;
and if the target data is less than or equal to the preset capacity threshold value, storing the target data in the first storage pool.
In this embodiment, a large file is stored in multiple data pools, the data stored in the first storage pool is smaller than the data in the second storage pool, and a small file is directly stored in the first storage pool, so that the data migrated by the file is very small, the migration rate is fast, and errors are not easy to occur. As for the preset capacity threshold, it can be decided according to the distributed file storage system and the operation service, and the value is used for determining whether the target data is a large file or a small file.
In order to make the technical solution of the present application more clear to those skilled in the art, the present application also provides an illustrative example in conjunction with fig. 2, where the first storage pool is a high-level pool, the second storage pool is a slow pool, the first 4M data of a file is written into the high-level pool, and the rest of the data is written into the slow pool, and the following contents may be included:
1. if the file only has 1 data pool, the migration client reads all data of the file from the first storage pool and writes the read data into the second storage pool; updating metadata to a metadata server MDS, namely modifying a data pool of a file from a first storage pool to a second storage pool, and finishing updating by the MDS on a response client; and after receiving the response, the client deletes the data in the first storage pool.
2. If the file records 2 data pools: reading the front 4M data of the file from the high-level pool, judging whether the reading is abnormal or not according to the data reading response information, and clearing the front 4M junk data of the file in the low-speed pool if the reading is abnormal; if the reading is successful, writing 4M data into the slow pool, judging whether the writing is abnormal according to data writing response information, and if the writing is abnormal, cleaning the front 4M junk data of the file in the slow pool; if the write is normal, the metadata is updated. The MDS responds to the migration client after updating, and if the metadata modification fails, the front 4M junk data of the file in the slow pool are cleared; if the metadata modification is successful, the 4M data in the high-level pool is deleted. And the migration client deletes the front 4M data of the file in the high-level pool, and waits for the next time of cleaning the front 4M junk data of the file in the high-level pool if the deletion fails.
It can be known from the above that, in this embodiment, a file hierarchical large file data migration and exception handling method based on file multi-data pool storage is implemented, where the front 4M data of a large file only exists in a high-level pool, the rest of the large file is stored in a low-speed pool, and 4 steps of layered reading, writing, metadata modification and old data deletion are adapted respectively, that is, the data operated in each step is only 4M, and is not all data of the file, and when each stage is abnormal, only 4M junk data is cleaned, so that file hierarchical migration scenes are further enriched, system stability is improved, and competitiveness of products is improved.
It should be noted that, in the present application, there is no strict sequential execution order among the steps, and as long as a logical order is met, the steps may be executed simultaneously or according to a certain preset order, and fig. 1 to fig. 2 are only schematic manners, and do not represent only such an execution order.
The embodiment of the invention also provides a corresponding device for the data migration method, thereby further ensuring that the method has higher practicability. Wherein the means can be described separately from the functional module point of view and the hardware point of view. In the following, the data migration apparatus provided by the embodiment of the present invention is introduced, and the data migration apparatus described below and the data migration method described above may be referred to correspondingly.
Based on the angle of the functional module, referring to fig. 3, fig. 3 is a structural diagram of a data migration apparatus according to an embodiment of the present invention, in a specific implementation manner, the apparatus may include:
the storage pool construction module 301 is configured to pre-construct a first storage pool and a second storage pool having different storage performances, wherein the storage performance of the first storage pool is better than that of the second storage pool.
A data partitioning storage module 302, configured to partition target data into first sub data and second sub data, where the first sub data is stored in a first storage pool, and the second sub data is stored in a second storage pool;
a data migration module 303, configured to, when a migration instruction of the target data is received, write the first sub data read from the first storage pool into the second storage pool, and send a request for updating storage location information of the target data to the metadata server;
and a data deleting module 304, configured to delete the first sub-data from the first storage pool when the metadata server responds that the updating is completed, so as to complete the migration operation of the target data.
Optionally, in some embodiments of this embodiment, the data migration module 303 may be configured to: reading first subdata from a first storage pool; judging whether the first subdata is successfully read from the first storage pool or not; if the first subdata is successfully read from the first storage pool, writing the first subdata into the second storage pool; and if the first subdata is not successfully read from the first storage pool, deleting the junk data in the second storage pool and simultaneously sending migration failure information.
As an optional implementation manner of this embodiment, the apparatus may further include a deleting module, configured to delete the garbage data in the second storage pool and send migration failure information if the first sub data is not successfully written into the second storage pool; if the metadata server does not successfully update the metadata, deleting the junk data in the second storage pool, and sending migration failure information; and if the metadata server successfully updates the metadata, deleting the junk data in the first storage pool.
As another optional implementation manner of this embodiment, the apparatus may further include a post-migration processing module, for example, configured to, if the data deletion operation in the first storage pool is successfully completed, successfully perform the data migration; and if the data deleting operation in the first storage pool is not completed successfully, the data migration fails, and the junk data in the first storage pool is continuously deleted when the data migration operation is executed next time.
Optionally, in other embodiments of this embodiment, the data division and storage module 302 may be further configured to: when target data to be written is received, if the target data is larger than a preset capacity threshold, dividing the target data into first subdata and second subdata; the space occupation capacity value of the first subdata is smaller than that of the second subdata; and if the target data is less than or equal to the preset capacity threshold value, storing the target data in the first storage pool.
As an optional implementation manner of this embodiment, the data migration module 303 may further be configured to: an instruction is sent to the metadata server to update the data pool to the second storage pool.
The functions of each functional module of the data migration apparatus in the embodiment of the present invention may be specifically implemented according to the method in the foregoing method embodiment, and the specific implementation process may refer to the description related to the foregoing method embodiment, which is not described herein again.
Therefore, the file migration error probability is reduced on the basis of improving the file migration efficiency.
The data migration apparatus mentioned above is described from the perspective of the functional module, and further, the present application also provides an electronic device, which is described from the perspective of hardware. Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 4, the electronic device includes a memory 40 for storing a computer program; a processor 41, configured to implement the steps of the data migration method according to any of the above embodiments when executing the computer program.
The processor 41 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 41 may also be a controller, a microcontroller, a microprocessor or other data processing chip, and the like. The processor 41 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 41 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 41 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that the display screen needs to display. In some embodiments, processor 41 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.
Memory 40 may include one or more computer-readable storage media, which may be non-transitory. Memory 40 may also include high speed random access memory as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. The memory 40 may in some embodiments be an internal storage unit of the electronic device, for example a hard disk of a server. The memory 40 may also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk provided on a server, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Further, the memory 40 may also include both an internal storage unit and an external storage device of the electronic device. The memory 40 can be used for storing application software installed in the electronic device and various data, such as: the code of the program that executes the vulnerability handling method, etc. may also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 40 is at least used for storing the following computer program 401, wherein after being loaded and executed by the processor 41, the computer program can implement the relevant steps of the data migration method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 40 may also include an operating system 402, data 403, and the like, and the storage manner may be a transient storage or a permanent storage. Operating system 402 may include, among other things, Windows, Unix, Linux, and the like. The data 403 may include, but is not limited to, data corresponding to data migration results, and the like.
In some embodiments, the electronic device may further include a display 42, an input/output interface 43, a communication interface 44, alternatively referred to as a network interface, a power supply 45, and a communication bus 46. The display 42 and the input/output interface 43, such as a Keyboard (Keyboard), belong to a user interface, and the optional user interface may also include a standard wired interface, a wireless interface, and the like. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, as appropriate, is used for displaying information processed in the electronic device and for displaying a visualized user interface. The communication interface 44 may optionally include a wired interface and/or a wireless interface, such as a WI-FI interface, a bluetooth interface, etc., typically used to establish a communication connection between an electronic device and other electronic devices. The communication bus 46 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 4, but this does not indicate only one bus or one type of bus.
Those skilled in the art will appreciate that the configuration shown in fig. 4 is not intended to be limiting of the electronic device and may include more or fewer components than those shown, such as sensors 47, for example, to perform various functions.
The functions of the functional modules of the electronic device according to the embodiments of the present invention may be specifically implemented according to the method in the above method embodiments, and the specific implementation process may refer to the description related to the above method embodiments, which is not described herein again.
Therefore, the file migration error probability is reduced on the basis of improving the file migration efficiency.
It is to be understood that, if the data migration method in the above embodiments is implemented in the form of software functional units and sold or used as a stand-alone product, it may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application may be substantially or partially implemented in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the methods of the embodiments of the present application, or all or part of the technical solutions. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a multimedia card, a card type Memory (e.g., SD or DX Memory, etc.), a magnetic Memory, a removable magnetic disk, a CD-ROM, a magnetic or optical disk, and other various media capable of storing program codes.
Based on this, the embodiment of the present invention further provides a readable storage medium, which stores a computer program, and the computer program is executed by a processor, and the steps of the data migration method according to any one of the above embodiments are provided.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. For hardware including devices and electronic equipment disclosed by the embodiment, the description is relatively simple because the hardware includes the devices and the electronic equipment correspond to the method disclosed by the embodiment, and the relevant points can be obtained by referring to the description of the method.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The data migration method, the data migration apparatus, the electronic device, and the readable storage medium provided by the present application are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A method of data migration, comprising:
pre-constructing a first storage pool and a second storage pool which have different storage performances, wherein the storage performance of the first storage pool is superior to that of the second storage pool;
dividing target data into first subdata and second subdata; the first subdata is stored in the first storage pool, and the second subdata is stored in the second storage pool;
when a migration instruction of the target data is received, writing the first subdata read from the first storage pool into the second storage pool, and simultaneously sending a request for updating storage position information of the target data to a metadata server;
and when the metadata server responds to the completion of updating, deleting the first subdata from the first storage pool so as to complete the migration operation of the target data.
2. The data migration method of claim 1, wherein said writing said first sub-data read from said first storage pool to said second storage pool comprises:
reading the first subdata from the first storage pool;
judging whether the first subdata is successfully read from the first storage pool or not;
if the first subdata is successfully read from the first storage pool, writing the first subdata into the second storage pool;
and if the first subdata is not successfully read from the first storage pool, deleting the junk data in the second storage pool and simultaneously sending migration failure information.
3. The data migration method of claim 1, wherein after writing the first sub-data read from the first storage pool to the second storage pool, comprising:
judging whether the first subdata is successfully written into the second storage pool or not;
and if the first subdata is not successfully written into the second storage pool, deleting the junk data in the second storage pool, and simultaneously sending migration failure information.
4. The data migration method according to claim 1, wherein after sending the request for updating the storage location information of the target data to the metadata server, the method further comprises:
judging whether the metadata server successfully updates the metadata;
if the metadata server does not successfully update the metadata, deleting the junk data in the second storage pool, and sending migration failure information;
and if the metadata server successfully updates the metadata, deleting the junk data in the first storage pool.
5. The data migration method of claim 4, wherein after said deleting garbage data from said first storage pool, further comprising:
if the data deleting operation in the first storage pool is successfully completed, the data migration is successful;
and if the data deleting operation in the first storage pool is not completed successfully, the data migration fails, and the junk data in the first storage pool is continuously deleted when the data migration operation is executed next time.
6. The data migration method according to any one of claims 1 to 5, wherein the dividing the target data into the first sub-data and the second sub-data includes:
when target data to be written is received, judging whether the target data is larger than a preset capacity threshold value;
if the target data is larger than a preset capacity threshold, dividing the target data into first subdata and second subdata; the space occupation capacity value of the first subdata is smaller than that of the second subdata;
and if the target data is less than or equal to a preset capacity threshold value, storing the target data to the first storage pool.
7. The data migration method according to claim 6, wherein said sending a request to a metadata server to update storage location information of the target data comprises:
and sending an instruction for updating the data pool into the second storage pool to the metadata server.
8. A data migration apparatus, comprising:
the storage pool constructing module is used for constructing a first storage pool and a second storage pool with different storage performances in advance, wherein the storage performance of the first storage pool is superior to that of the second storage pool;
the data segmentation storage module is used for segmenting target data into first subdata and second subdata; the first subdata is stored in the first storage pool, and the second subdata is stored in the second storage pool;
a data migration module, configured to, when receiving a migration instruction of the target data, write the first sub-data read from the first storage pool into the second storage pool, and send a request for updating storage location information of the target data to a metadata server;
and the data deleting module is used for deleting the first subdata from the first storage pool when the metadata server responds that the updating is completed so as to complete the migration operation of the target data.
9. An electronic device comprising a processor and a memory, the processor being configured to implement the steps of the data migration method according to any one of claims 1 to 7 when executing a computer program stored in the memory.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the data migration method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111149417.0A CN113836116A (en) | 2021-09-29 | 2021-09-29 | Data migration method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111149417.0A CN113836116A (en) | 2021-09-29 | 2021-09-29 | Data migration method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113836116A true CN113836116A (en) | 2021-12-24 |
Family
ID=78967506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111149417.0A Pending CN113836116A (en) | 2021-09-29 | 2021-09-29 | Data migration method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113836116A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114936003A (en) * | 2022-05-06 | 2022-08-23 | 北京新科安云信息技术有限公司 | Data layered migration method, device and equipment of resource pool and readable storage medium |
| WO2023241115A1 (en) * | 2022-06-13 | 2023-12-21 | 华为技术有限公司 | Data migration method and related apparatus |
-
2021
- 2021-09-29 CN CN202111149417.0A patent/CN113836116A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114936003A (en) * | 2022-05-06 | 2022-08-23 | 北京新科安云信息技术有限公司 | Data layered migration method, device and equipment of resource pool and readable storage medium |
| CN114936003B (en) * | 2022-05-06 | 2023-03-21 | 北京新科安云信息技术有限公司 | Data layered migration method, device and equipment of resource pool and readable storage medium |
| WO2023241115A1 (en) * | 2022-06-13 | 2023-12-21 | 华为技术有限公司 | Data migration method and related apparatus |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106201659B (en) | A kind of method and host of live migration of virtual machine | |
| CN109445861B (en) | System starting method, device, computer device and storage medium | |
| CN105468473A (en) | Data migration method and data migration apparatus | |
| CN113836116A (en) | Data migration method and device, electronic equipment and readable storage medium | |
| CN103617097B (en) | File access pattern method and device | |
| CN107656939A (en) | File wiring method and device | |
| CN102938784A (en) | Method and system used for data storage and used in distributed storage system | |
| US10489289B1 (en) | Physical media aware spacially coupled journaling and trim | |
| CN112000426B (en) | Data processing method and device | |
| KR20210058118A (en) | Casedb: low-cost put-intensive key-value store for edge computing | |
| CN108763511B (en) | Layer layout method and device in page, electronic equipment and storage medium | |
| CN113568582A (en) | Data management method and device and storage equipment | |
| CN104268097B (en) | A kind of metadata processing method and system | |
| CN110780814B (en) | Stored data sorting method and device | |
| CN107329836B (en) | Multi-system memory management method and device and mobile terminal | |
| CN117369731B (en) | Data reduction processing method, device, equipment and medium | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| US10142415B2 (en) | Data migration | |
| CN112148217A (en) | Caching method, device and medium for deduplication metadata of full flash storage system | |
| CN113687779B (en) | Data migration method, device, electronic equipment and readable storage medium | |
| CN108763341A (en) | Electronic device, automatic Building table method and storage medium | |
| CN113821302A (en) | Method and device for deploying virtual machine on virtualization cloud platform, electronic equipment and medium | |
| CN108205559B (en) | Data management method and equipment thereof | |
| CN109189724A (en) | Improve the method and device of video monitoring system audio, video data storage efficiency | |
| CN103970671B (en) | Allocating Additional Requested Storage Space For A Data Set In A First Managed Space In A Second Managed Space |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |