CN116737088B - Object migration method and device, electronic equipment and storage medium - Google Patents
Object migration method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116737088B CN116737088B CN202311021679.8A CN202311021679A CN116737088B CN 116737088 B CN116737088 B CN 116737088B CN 202311021679 A CN202311021679 A CN 202311021679A CN 116737088 B CN116737088 B CN 116737088B
- Authority
- CN
- China
- Prior art keywords
- migration
- migrated
- transmission rate
- objects
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005012 migration Effects 0.000 title claims abstract description 216
- 238000013508 migration Methods 0.000 title claims abstract description 216
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000005540 biological transmission Effects 0.000 claims abstract description 113
- 238000012545 processing Methods 0.000 claims description 25
- 230000000903 blocking effect Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 description 15
- 238000005192 partition Methods 0.000 description 9
- 238000012360 testing method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 239000002071 nanotube Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an object migration method, an object migration device, electronic equipment and a storage medium, and relates to the technical field of cloud computing, wherein the method comprises the following steps: dividing the objects in the object list to be migrated into a first object and a second object based on a target transmission rate, wherein the target transmission rate is the transmission rate of the objects migrated from a source end to a target end, and the byte number of the first object is larger than that of the second object; migrating the first object based on a first migration policy; and migrating the second object based on the second migration policy. The method comprises the steps of obtaining a target transmission rate in a current migration environment, dividing objects in an object list to be migrated into a first object and a second object based on the target transmission rate, and then performing object migration by adopting a first migration strategy and a second migration strategy respectively for the first object and the second object. The hardware resources in the current migration environment are fully utilized, so that the efficiency of migrating the object to be migrated from the source end to the target end is improved.
Description
Technical Field
The present invention relates to the field of cloud computing technologies, and in particular, to an object migration method, an object migration device, an electronic device, and a storage medium.
Background
The object storage service is widely applied to the fields of cloud computing or cloud storage, and the like, and the object storage data migration generally adopts a data disk-falling mode for migration or directly carries out data stream transmission through a memory. In the traditional object storage data migration mode, the export and import of a large amount of files occupy a large space capacity, and an additional storage device for transferring storage objects is required to be provided outside a source object storage cluster and a target object storage cluster; the memory-based object storage migration method simply obtains the object list and performs data migration of all objects in the bucket, and does not improve the data transmission rate of each thread.
It can be seen that the prior art has the problem of poor object migration efficiency.
Disclosure of Invention
The embodiment of the invention provides an object migration method, an object migration device, electronic equipment and a storage medium, which are used for solving the problem of poor object migration efficiency in the prior art.
In order to solve the technical problems, the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides an object migration method, which is used for a terminal, where the method includes:
dividing objects in an object list to be migrated into a first object and a second object based on a target transmission rate, wherein the target transmission rate is the transmission rate of the objects to be migrated from a source end to a target end, and the byte number of the first object is larger than that of the second object;
Migrating the first object based on a first migration policy;
and migrating the second object based on a second migration policy.
Optionally, in a case that the number of the first objects is greater than or equal to M, M is an integer greater than 1, the migrating the first object based on the first migration policy includes:
determining a first thread number based on the byte number of the first object and the target transmission rate;
determining a second thread number based on the CPU core number of the central processing unit of the terminal;
determining the one of the first thread number and the second thread number with the smaller thread number as the target thread number;
and simultaneously migrating M first objects, wherein each first object is migrated by adopting the target line number.
Optionally, in a case that the number of the second objects is greater than or equal to N, N is the number of CPU cores of the terminal, and N is an integer greater than 1, the migrating the second object based on the second migration policy includes:
and simultaneously migrating the N second objects, wherein each second object is migrated by adopting a preset thread number.
Optionally, the dividing the objects in the object list to be migrated into the first object and the second object based on the target transmission rate includes:
Determining an object with the byte number larger than K times of the target transmission rate in the object list to be migrated as a first object;
determining an object with the byte number smaller than or equal to K times the target transmission rate in the object list to be migrated as a second object;
wherein K is an integer greater than 0.
Optionally, said each of said first objects migrates using said target number of threads, including:
performing block processing on the first object based on the target transmission rate;
and migrating the first object after the blocking by adopting the target line number.
In a second aspect, an embodiment of the present invention further provides an object migration apparatus, where the apparatus includes:
the dividing module is used for dividing the objects in the object list to be migrated into a first object and a second object based on a target transmission rate, wherein the target transmission rate is the transmission rate of the objects migrated from the source end to the target end, and the byte number of the first object is larger than that of the second object;
the first migration module is used for migrating the first object based on a first migration policy;
and the second migration module is used for migrating the second object based on a second migration policy.
Optionally, in a case where the number of the first objects is greater than or equal to M, M is an integer greater than 1, the first migration module includes:
a first determination submodule for determining a first thread number based on the byte number of the first object and the target transmission rate;
the second determining submodule is used for determining a second thread number based on the CPU core number of the central processing unit of the terminal;
a third determination submodule for determining one of the first thread number and the second thread number, which is smaller in thread number, as a target thread number;
and the first migration submodule is used for simultaneously migrating M first objects, and each first object adopts the target line number for migration.
Optionally, in the case that the number of the second objects is greater than or equal to N, N is the number of CPU cores of the terminal, and N is an integer greater than 1, the second migration module includes:
and the second migration submodule is used for simultaneously migrating the N second objects, and each second object adopts a preset thread number for migration.
Optionally, the dividing module includes:
a fourth determining submodule, configured to determine an object whose number of bytes in the object list to be migrated is greater than K times the target transmission rate as a first object;
A fifth determining submodule, configured to determine an object with a byte number less than or equal to K times the target transmission rate in the object list to be migrated as a second object;
wherein K is an integer greater than 0.
Optionally, the first migration submodule includes:
the blocking processing unit is used for carrying out blocking processing on the first object based on the target transmission rate;
and the migration unit is used for migrating the segmented first object by adopting the target line number.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor,
the processor is configured to divide an object in the object list to be migrated into a first object and a second object based on a target transmission rate, where the target transmission rate is a transmission rate of the object migrated from the source end to the target end, and a byte number of the first object is greater than a byte number of the second object;
the processor is further configured to migrate the first object based on a first migration policy;
the processor is further configured to migrate the second object based on a second migration policy.
Optionally, in a case that the number of the first objects is greater than or equal to M, M is an integer greater than 1, the migrating the first object based on the first migration policy includes:
Determining a first thread number based on the byte number of the first object and the target transmission rate;
determining a second thread number based on the CPU core number of the central processing unit of the terminal;
determining the one of the first thread number and the second thread number with the smaller thread number as the target thread number;
and simultaneously migrating M first objects, wherein each first object is migrated by adopting the target line number.
Optionally, in a case that the number of the second objects is greater than or equal to N, N is the number of CPU cores of the terminal, and N is an integer greater than 1, the migrating the second object based on the second migration policy includes:
and simultaneously migrating the N second objects, wherein each second object is migrated by adopting a preset thread number.
Optionally, the dividing the objects in the object list to be migrated into the first object and the second object based on the target transmission rate includes:
determining an object with the byte number larger than K times of the target transmission rate in the object list to be migrated as a first object;
determining an object with the byte number smaller than or equal to K times the target transmission rate in the object list to be migrated as a second object;
Wherein K is an integer greater than 0.
Optionally, said each of said first objects migrates using said target number of threads, including:
performing block processing on the first object based on the target transmission rate;
and migrating the first object after the blocking by adopting the target line number.
In a fourth aspect, an embodiment of the present invention further provides an electronic device, including: the object migration method comprises a processor, a memory and a program stored in the memory and capable of running on the processor, wherein the program is executed by the processor to realize the steps of the object migration method.
In a fifth aspect, embodiments of the present invention further provide a computer readable storage medium having a computer program stored thereon, the computer program implementing the steps of the object migration method described above when executed by a processor.
In the embodiment of the invention, the object in the object list to be migrated is divided into the first object and the second object by acquiring the target transmission rate in the current migration environment and taking the target transmission rate as a reference, and then the first migration strategy and the second migration strategy are adopted for the first object and the second object respectively for object migration. By means of different migration strategies, hardware resources in the current migration environment are fully utilized, and therefore efficiency of migration of the object to be migrated from the source end to the target end is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is one of the flowcharts of an object migration method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an object migration apparatus according to an embodiment of the present invention;
FIG. 3 is a second flowchart of an object migration method according to an embodiment of the present invention;
FIG. 4 is a second schematic diagram of an object migration apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, fig. 1 is one of flowcharts of an object migration method provided by an embodiment of the present invention, which is used for a terminal, and as shown in fig. 1, the method includes the following steps:
step 101, dividing objects in an object list to be migrated into a first object and a second object based on a target transmission rate, wherein the target transmission rate is a transmission rate of objects migrated from a source end to a target end, and the byte number of the first object is larger than that of the second object;
the terminal can be a mobile phone, a computer, a local server and other devices. In the case that the source terminal is a terminal, the target terminal may be a cloud terminal; in the case where the target terminal is a terminal, the source terminal may be a cloud terminal. The cloud may be a cloud server, a cloud storage gateway, or the like. The target transmission rate may be a maximum transmission rate at which the object migrates from the source end to the target end.
In an example, the target transmission rate in the current environment may be obtained by obtaining the maximum transmission rate at which the test object migrates from the source end to the target end. The objects in the object list to be migrated are divided based on the target transmission rate to obtain a first object with a large number of bytes (i.e. a large file) and a second object with a small number of bytes (i.e. a small file). Under the migration environment of different hardware resources, for example, larger bandwidth, more processors, larger memory, etc., the obtained target transmission rate will also be correspondingly higher. Therefore, when dividing the objects in the object list to be migrated, the target transmission rate in the current environment is considered, and the accuracy of object division is improved. And different migration strategies are convenient to carry out on objects with different sizes in the follow-up process.
Where the basic unit of object migration/storage is an object, an object is actually a combination of data of a file and a set of attribute information (i.e., metadata), and typically an object generally belongs to a certain bucket. Based on the obtained maximum transmission rate as a reference, dividing the migration task of each bucket into a first object migration subtask and a second object migration subtask according to the size of the object to be migrated at the source end. Through step 102 and step 103, specific parameter tuning is performed for different subtasks, and meanwhile, the tuning needs to be combined with the resource configuration of the current migration environment, so that the migration rate is maximized under the condition that no additional resource consumption is increased, and the object migration efficiency is greatly improved.
Step 102, migrating the first object based on a first migration policy;
the first object may be an object with a larger number of bytes in the object list to be migrated, i.e. a large file; the first migration policy may include a policy for increasing the number of threads corresponding to the object to be migrated. The efficiency of the migration of the first objects from the source end to the target end can be improved by increasing the thread number of each first object during the migration.
And step 103, migrating the second object based on a second migration policy.
The second object may be an object with a smaller number of bytes in the object list to be migrated, i.e. a small file; the second migration policy includes a policy for increasing the number of concurrencies corresponding to the object to be migrated. The efficiency of migrating the second object from the source end to the target end can be improved by increasing the concurrency number of migrating the second object.
In this embodiment, the object in the object list to be migrated is divided into a first object and a second object by acquiring a target transmission rate in the current migration environment and based on the target transmission rate as a reference, and then the first migration policy and the second migration policy are adopted for the first object and the second object, respectively, to migrate the object. By means of different migration strategies, hardware resources in the current migration environment are fully utilized, and therefore efficiency of migration of the object to be migrated from the source end to the target end is improved.
In some optional embodiments, the dividing the objects in the object list to be migrated into the first object and the second object based on the target transmission rate includes:
determining an object with the byte number larger than K times of the target transmission rate in the object list to be migrated as a first object;
determining an object with the byte number smaller than or equal to K times the target transmission rate in the object list to be migrated as a second object;
Wherein K is an integer greater than 0.
In an example, the value of K may be 4. Specifically, a test object can be selected and migrated from a source end to a target end, the transmission rate of the test object is recorded in real time in the migration process, and the maximum transmission rate of the test object migrated from the source end to the target end is used as the target transmission rate of the current migration environment. In addition, the target transmission rate of the current migration environment can be obtained through information such as bandwidth and the like based on the hardware resources of the current migration environment. The target transfer rate of the current migration environment may be denoted as X MB/s.
Then, based on the value of X, determining an object with the byte number larger than 4X X MB in the object list to be migrated as a first object; and determining an object with the byte number smaller than or equal to 4 x MB in the object list to be migrated as a second object. In this way, all object lists to be migrated are divided into two lists of the size file, so that different migration strategies can be conveniently carried out on objects with different sizes in the follow-up process.
It should be understood that, in other examples, K is an integer greater than 0, and the value of K may also be other values, for example, K is equal to 1, 2, 3, 5, 10 or 20, etc., according to the target transmission rate and the number of bytes of the object to be migrated, which will not be described herein.
In some optional embodiments, where the number of the first objects is greater than or equal to M, M is an integer greater than 1, the migrating the first object based on the first migration policy includes:
determining a first thread number based on the byte number of the first object and the target transmission rate;
determining a second thread number based on a central processing unit (Central Processing Unit, CPU) core number of the terminal;
determining the one of the first thread number and the second thread number with the smaller thread number as the target thread number;
and simultaneously migrating M first objects, wherein each first object is migrated by adopting the target line number.
In an example, objects in a to-be-migrated object list are divided based on a target transmission rate, and after a first object with a large number of bytes and a second object with a small number of bytes are obtained, the first object is migrated based on a first migration policy. The CPU core number of the current migration environment is greater than or equal to 2, and when the number of the first objects is greater than or equal to M, the second thread number is determined to be C/2 based on the CPU core number C; determining a first thread number as Z/X based on the byte number Z of the first object and the target transmission rate X; the target thread number is one of the first thread number and the second thread number, i.e., the target thread number is: min (C/2, Z/X). In this way, M first objects are migrated at the same time, and each first object is migrated by using min (C/2, Z/X) threads. When a thread used for migration is allocated for each first object, resources such as a CPU (Central processing Unit) are considered, hardware resources in the current migration environment are fully utilized, and the situation that the migration rate is reduced due to thread competition is avoided; meanwhile, the smaller thread number is adopted as the target thread number in the first thread number and the second thread number, so that the influence of block switching can be reduced, and the efficiency of the first object moving from the source end to the target end is improved.
Specifically, too many or too few threads are configured in the same CPU core, which tends to reduce the migration rate. Thus, two threads may be configured for one CPU core to enhance the migration rate of objects.
Assuming that the CPU core number C is 8, the value of m is 4, the target transfer rate may be X, and the byte number Z of the first object may be 5X. In case the number of first objects is smaller than 4, the first objects smaller than 4 may be migrated at the same time; in the case where the number of first objects is greater than or equal to 4, migration may be performed on 4 first objects at the same time. The number of target threads adopted in the migration of each first object is as follows: min (8/2,5X/X) =4. The CPU and other resources are considered, the hardware resources in the current migration environment are fully utilized, the situation that the migration rate is reduced due to thread competition is avoided, and the efficiency of migrating the first object from the source end to the target end is improved.
Assuming that the CPU core number C is 16 and the value of m is 4, the target transmission rate may be X. In the case where the number of first objects is 7, 4 first objects among the 7 first objects may be migrated at the same time. Of the 4 first objects: the number of bytes Z of the first object a may be 5X, the number of bytes Z of the first object B may be 6X, the number of bytes Z of the first object C may be 10X, and the number of bytes Z of the first object D may be 7X. When the first object a, the first object B, the first object C and the first object D are migrated at the same time, the target number of threads adopted by the first object a is: min (16/2,5X/X) =5; the target number of runs adopted by the first object B is: min (16/2,6X/X) =6; the target number of runs adopted by the first object C is: min (16/2, 10X/X) =8; the target number of passes taken by the first object D is: min (16/2,7X/X) =7. The remaining 3 first objects may be cached in the queue first, and after the migration of the first object a, the first object B, the first object C, and the first object is completed, the remaining 3 first objects are migrated at the same time. In this way, by adopting the smaller thread number of the first thread number and the second thread number as the target thread number, the influence of block switching is reduced, and the efficiency of migrating the first object from the source end to the target end is improved.
In some optional embodiments, in a case where the number of the second objects is greater than or equal to N, N is a CPU core number of the terminal, and N is an integer greater than 1, the migrating the second object based on the second migration policy includes:
and simultaneously migrating the N second objects, wherein each second object is migrated by adopting a preset thread number.
In an example, the objects in the object list to be migrated are divided based on the target transmission rate, and after a first object with a larger number of bytes and a second object with a smaller number of bytes are obtained, the second object is migrated based on a second migration policy. And the CPU core number of the current migration environment is greater than or equal to 2, and when the number of the second objects is greater than or equal to N, N second objects are migrated at the same time, and each second object is migrated by adopting a preset thread number. Wherein, the preset thread number can be 2; n is the CPU core number of the terminal, namely N is more than or equal to 2. Therefore, the concurrency number of the second object is increased, frequent switching after the object migration is completed is avoided, and therefore the efficiency of the second object from the source end to the target end is improved.
Specifically, too many or too few threads are configured in the same CPU core, which tends to reduce the migration rate. Thus, two threads may be configured for one CPU core to enhance the migration rate of objects.
Assuming that the number of CPU cores C is 8, the value of N is 8, the target transmission rate can be X, and under the condition that the number of second objects is smaller than 8, the second objects smaller than 8 can be migrated at the same time; in the case that the number of the second objects is greater than or equal to 8, migration can be performed on 8 first objects at the same time, and the number of threads adopted in migration of each second object is 2.
Let the number of CPU cores C be 16 and the value of n be 16. In the case where the number of second objects is 8, the 8 second objects may be migrated simultaneously, and each second object is migrated using 2 threads. Under the condition that the number of the second objects is 20, 16 second objects can be migrated at the same time, each second object is migrated by adopting 2 threads, the rest 4 second objects can be cached in the queue first, and after the 16 second objects are migrated, the rest 4 second objects are migrated at the same time. Therefore, the concurrency number of the second object is increased, frequent switching after the object migration is completed is avoided, and therefore the efficiency of the second object from the source end to the target end is improved.
All larger objects (i.e., first objects) in the object migration can be completed by partition migration, and the smaller objects (i.e., second objects) can be also completed by one or two partitions, wherein the size and the number of the partitions have a certain influence on the migration rate.
In some alternative embodiments, said each of said first objects is migrated using said target number of threads, comprising:
performing block processing on the first object based on the target transmission rate;
and migrating the first object after the blocking by adopting the target line number.
Specifically, the maximum rate of migration per unit time of the current environment is 10MB/S, that is, the target transmission rate is 10MB/S, the default chunk size may be set to 10MB, so that the chunk size matches the maximum transmission rate, and migration is performed on the first object after the chunk processing based on the default chunk, for example, when the number of bytes of the second object is 500MB, 50 chunks need to be allocated. Threads of a target thread number may be configured to migrate the partitions of each object to achieve optimal migration performance. When the block size is set to be smaller, such as 1MB, 500 blocks exist, the migration time of each block is very short, and a large number of switching actions can greatly reduce the migration rate; on the contrary, when the setting of the partition is larger, on one hand, the object storage itself has a limitation on the size of the object, on the other hand, the larger partition can cause higher memory occupation, in addition, the probability of abnormal data stream transmission of the larger partition in the migration process is greatly improved, and the retransmission cost after abnormal transmission of each partition is also larger. When a second object with the size of 1MB is migrated, only one partition is needed to complete the migration of the whole object, so when a traditional thread pool is used, the configuration is based on an object list to perform concurrent migration, and small files can be instantly migrated, and the situation that the whole physical resource is not full can occur due to frequent switching of the migrated objects, so that the strategy of simultaneously migrating more objects can ensure the whole migration efficiency. Through actual testing (same migration environment and migration object), the test results of the effect of improving the migration rate are shown in the following table 1:
TABLE 1
As can be seen from table 1, the first migration policy and the second migration policy are both comprehensively matched with the maximum transmission rate, the byte number of the object to be migrated and the physical resources in the current migration environment, so as to fully utilize the physical resources in the current migration environment, thereby improving the efficiency of migrating the object to be migrated from the source end to the target end.
The embodiment of the invention provides an object migration method, which improves the object migration rate under the condition of the existing resources, as shown in fig. 2, the maximum migration rate (namely, the target transmission rate) under the current environment can be obtained through an object detection unit in an object migration device, and the block size of an object in the migration process is matched with the transmission rate. After the objects in the object list to be migrated are divided into a first object and a second object based on the target transmission rate, the first object can be migrated through the large file migration unit based on a first migration strategy; the second object can be migrated through the doclet migration unit and based on the second migration policy.
Specifically, as shown in fig. 3, the main process of the object migration method is as follows:
1) Obtaining the maximum transmission rate X MB/s from a source end to a target end of the current migration environment;
2) Dividing a file list of a barrel to be migrated;
2.1 Pulling all object lists to be migrated;
2.2 Determining an object with the byte number larger than 4X MB in the object list to be migrated as a first object (i.e. a large file) based on the value of X; determining an object with the byte number smaller than or equal to 4 x MB in the object list to be migrated as a second object (namely a small file), and dividing all the object list to be migrated into two lists of a size file;
3) Selecting different migration strategies for different file sizes;
assuming that the CPU core number of each migration environment is C (C > =2), performing independent migration policy optimization for two parts of a large file and a small file, wherein the method mainly comprises the following three parts:
3.1 The optimization parameters contained in the migration module comprise the block size, the number of the migration objects and the number of threads adopted by each migration object;
3.2 X MB is used as the default block size in the first object migration process, 4 first object simultaneous migration strategies are adopted under the assumption that the large file object size is Z MB, and specifically, each first object migration adopts min (C/2, Z/X) threads for migration. When the thread used for migration is distributed for each first object, resources such as a CPU (Central processing Unit) are considered, hardware resources in the current migration environment are fully utilized, and the situation that the migration rate is reduced due to thread competition is avoided; meanwhile, the smaller thread number is adopted as the target thread number in the first thread number and the second thread number, so that the influence of block switching can be reduced, and the efficiency of the first object moving from the source end to the target end is improved.
3.3 The second migration strategy adopts X MB as the default block size in the second object migration process, adopts C second object simultaneous migration strategies by default, and specifically adopts 2 threads for migration for each second object migration. Therefore, the concurrency number of the second object is increased, frequent switching after the object migration is completed is avoided, and therefore the efficiency of the second object from the source end to the target end is improved.
4) Performing data migration;
4.1 Generating a large-size object migration subtask according to a corresponding migration strategy by the large-size object list, distributing all machine resources by a migration module cluster unified nano tube, and synchronously starting a migration module to synchronously migrate the rest object list if one of the file lists is migrated until all objects are migrated;
4.2 Marking the file blocks with transmission errors in the transmission process, retransmitting the file blocks, and finally completing the data migration of all objects in the barrel;
5) Checking transmission data to finish migration;
5.1 During the process of object block transmission, calculating a Message-Digest Algorithm 5 (MD5) value for the memory data stream of each block to ensure the integrity of each block transmission;
5.2 After the migration of each object is completed, verifying the metadata of the object before and after the migration so as to ensure the consistency of the object before and after the migration;
5.3 Detecting that all objects in the large and small file migration list have been migrated to the target object bucket.
In this embodiment, the first migration policy and the second migration policy are both comprehensively matched with the maximum transmission rate in the current migration environment, the byte number of the object to be migrated and the physical resources, so as to fully utilize the physical resources in the current migration environment, thereby improving the efficiency of migrating the object to be migrated from the source end to the target end.
Referring to fig. 4, fig. 4 is a second schematic structural diagram of an object migration apparatus according to an embodiment of the present invention, and as shown in fig. 4, an object migration apparatus 400 includes:
the dividing module 401 is configured to divide an object in the object list to be migrated into a first object and a second object based on a target transmission rate, where the target transmission rate is a transmission rate of the object migrated from a source end to a target end, and a byte number of the first object is greater than a byte number of the second object;
a first migration module 402, configured to migrate the first object based on a first migration policy;
And the second migration module 403 is configured to migrate the second object based on a second migration policy.
Optionally, where the number of the first objects is greater than or equal to M, M is an integer greater than 1, the first migration module 402 includes:
a first determination submodule for determining a first thread number based on the byte number of the first object and the target transmission rate;
the second determining submodule is used for determining a second thread number based on the CPU core number of the central processing unit of the terminal;
a third determination submodule for determining one of the first thread number and the second thread number, which is smaller in thread number, as a target thread number;
and the first migration submodule is used for simultaneously migrating M first objects, and each first object adopts the target line number for migration.
Optionally, in the case where the number of the second objects is greater than or equal to N, N is the CPU core number of the terminal, and N is an integer greater than 1, the second migration module 403 includes:
and the second migration submodule is used for simultaneously migrating the N second objects, and each second object adopts a preset thread number for migration.
Optionally, the dividing module 401 includes:
a fourth determining submodule, configured to determine an object whose number of bytes in the object list to be migrated is greater than K times the target transmission rate as a first object;
a fifth determining submodule, configured to determine an object with a byte number less than or equal to K times the target transmission rate in the object list to be migrated as a second object;
wherein K is an integer greater than 0.
Optionally, the first migration submodule includes:
the blocking processing unit is used for carrying out blocking processing on the first object based on the target transmission rate;
and the migration unit is used for migrating the segmented first object by adopting the target line number.
It should be noted that, the object migration apparatus 400 can implement each process of each embodiment of the above object migration method, technical features are in one-to-one correspondence, and the same technical effects can be achieved, so that repetition is avoided, and detailed description is omitted here.
The embodiment of the invention also provides electronic equipment, which comprises: the processor, the memory and the program stored in the memory and capable of running on the processor, when the program is executed by the processor, the processes of the above embodiment of the object migration method are implemented, and the same technical effects can be achieved, so that repetition is avoided, and no description is repeated here.
Specifically, referring to fig. 5, an embodiment of the present invention further provides an electronic device, including a bus 501, a transceiver 502, an antenna 503, a bus interface 504, a processor 505, and a memory 506.
Wherein the transceiver 502 is configured to obtain a target transmission rate;
a processor 505, configured to divide objects in a to-be-migrated object list into a first object and a second object based on a target transmission rate, where the target transmission rate is a transmission rate of objects migrated from a source end to a target end, and a byte number of the first object is greater than a byte number of the second object;
the processor 505 is further configured to migrate the first object based on a first migration policy;
the processor 505 is further configured to migrate the second object based on a second migration policy.
Optionally, in a case that the number of the first objects is greater than or equal to M, M is an integer greater than 1, the migrating the first object based on the first migration policy includes:
determining a first thread number based on the byte number of the first object and the target transmission rate;
determining a second thread number based on the CPU core number of the central processing unit of the terminal;
determining the one of the first thread number and the second thread number with the smaller thread number as the target thread number;
And simultaneously migrating M first objects, wherein each first object is migrated by adopting the target line number.
Optionally, in a case that the number of the second objects is greater than or equal to N, N is the number of CPU cores of the terminal, and N is an integer greater than 1, the migrating the second object based on the second migration policy includes:
and simultaneously migrating the N second objects, wherein each second object is migrated by adopting a preset thread number.
Optionally, the dividing the objects in the object list to be migrated into the first object and the second object based on the target transmission rate includes:
determining an object with the byte number larger than K times of the target transmission rate in the object list to be migrated as a first object;
determining an object with the byte number smaller than or equal to K times the target transmission rate in the object list to be migrated as a second object;
wherein K is an integer greater than 0.
Optionally, said each of said first objects migrates using said target number of threads, including:
performing block processing on the first object based on the target transmission rate;
and migrating the first object after the blocking by adopting the target line number.
In fig. 5, a bus architecture (represented by bus 501), the bus 501 may include any number of interconnected buses and bridges, with the bus 501 linking together various circuits, including one or more processors, represented by processor 505, and memory, represented by memory 506. The bus 501 may also link together various other circuits such as peripheral devices, voltage regulators, power management circuits, etc., which are well known in the art and, therefore, will not be described further herein. Bus interface 504 provides an interface between bus 501 and transceiver 502. The transceiver 502 may be one element or may be multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 505 is transmitted over a wireless medium via the antenna 503, and further, the antenna 503 receives the data and transmits the data to the processor 505.
The processor 505 is responsible for managing the bus 501 and general processing, and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 506 may be used to store data used by processor 505 in performing operations.
Alternatively, the processor 505 may be a central processing unit (Central Processing Unit, CPU), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA), or a complex programmable logic device (Complex Programmable Logic Device, CPLD).
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the processes of the above-mentioned object migration method embodiment, and can achieve the same technical effects, so that repetition is avoided, and no further description is given here. Wherein the computer readable storage medium is selected from Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present invention is not limited to performing the functions in the order discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (6)
1. An object migration method, characterized by being used for a terminal, the method comprising:
dividing objects in an object list to be migrated into a first object and a second object based on a target transmission rate, wherein the target transmission rate is the transmission rate of the objects to be migrated from a source end to a target end, and the byte number of the first object is larger than that of the second object;
migrating the first object based on a first migration policy;
migrating the second object based on a second migration policy;
wherein, when the number of the first objects is greater than or equal to M, M is an integer greater than 1, and the migrating the first object based on the first migration policy includes:
determining a first thread number based on the byte number of the first object and the target transmission rate;
determining a second thread number based on the CPU core number of the central processing unit of the terminal;
determining the one of the first thread number and the second thread number with the smaller thread number as the target thread number;
simultaneously migrating M first objects, wherein each first object is migrated by adopting the target number of threads;
wherein, when the number of the second objects is greater than or equal to N, N is the CPU core number of the terminal, and N is an integer greater than 1, the migrating the second object based on the second migration policy includes:
Simultaneously migrating the N second objects, wherein each second object is migrated by adopting a preset thread number;
the dividing the objects in the object list to be migrated into a first object and a second object based on the target transmission rate includes:
determining an object with the byte number larger than K times of the target transmission rate in the object list to be migrated as a first object;
determining an object with the byte number smaller than or equal to K times the target transmission rate in the object list to be migrated as a second object;
wherein K is an integer greater than 0.
2. The method of claim 1, wherein each of the first objects migrates using the target number of threads, comprising:
performing block processing on the first object based on the target transmission rate;
and migrating the first object after the blocking by adopting the target line number.
3. An object migration apparatus, the apparatus comprising:
the dividing module is used for dividing the objects in the object list to be migrated into a first object and a second object based on a target transmission rate, wherein the target transmission rate is the transmission rate of the objects migrated from the source end to the target end, and the byte number of the first object is larger than that of the second object;
The first migration module is used for migrating the first object based on a first migration policy;
the second migration module is used for migrating the second object based on a second migration policy;
wherein, in the case that the number of the first objects is greater than or equal to M, M is an integer greater than 1, the first migration module includes:
a first determination submodule for determining a first thread number based on the byte number of the first object and the target transmission rate;
the second determining submodule is used for determining a second thread number based on the CPU core number of the central processing unit of the terminal;
a third determination submodule for determining one of the first thread number and the second thread number, which is smaller in thread number, as a target thread number;
the first migration submodule is used for simultaneously migrating M first objects, and each first object adopts the target line number for migration;
wherein, when the number of the second objects is greater than or equal to N, N is the CPU core number of the terminal, and N is an integer greater than 1, the second migration module includes:
the second migration submodule is used for simultaneously migrating N second objects, and each second object adopts a preset thread number for migration;
Wherein, the division module includes:
a fourth determining submodule, configured to determine an object whose number of bytes in the object list to be migrated is greater than K times the target transmission rate as a first object;
a fifth determining submodule, configured to determine an object with a byte number less than or equal to K times the target transmission rate in the object list to be migrated as a second object;
wherein K is an integer greater than 0.
4. An electronic device, comprising a processor,
the processor is configured to divide an object in the object list to be migrated into a first object and a second object based on a target transmission rate, where the target transmission rate is a transmission rate of the object migrated from the source end to the target end, and a byte number of the first object is greater than a byte number of the second object;
the processor is further configured to migrate the first object based on a first migration policy;
the processor is further configured to migrate the second object based on a second migration policy;
wherein, when the number of the first objects is greater than or equal to M, M is an integer greater than 1, and the migrating the first object based on the first migration policy includes:
Determining a first thread number based on the byte number of the first object and the target transmission rate;
determining a second thread number based on the CPU core number of the central processing unit of the terminal;
determining the one of the first thread number and the second thread number with the smaller thread number as the target thread number;
simultaneously migrating M first objects, wherein each first object is migrated by adopting the target number of threads;
wherein, when the number of the second objects is greater than or equal to N, N is the CPU core number of the terminal, and N is an integer greater than 1, the migrating the second object based on the second migration policy includes:
simultaneously migrating the N second objects, wherein each second object is migrated by adopting a preset thread number;
the dividing the objects in the object list to be migrated into a first object and a second object based on the target transmission rate includes:
determining an object with the byte number larger than K times of the target transmission rate in the object list to be migrated as a first object;
determining an object with the byte number smaller than or equal to K times the target transmission rate in the object list to be migrated as a second object;
Wherein K is an integer greater than 0.
5. An electronic device, comprising: a processor, a memory and a program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the object migration method according to any one of claims 1 to 2.
6. A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the object migration method according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311021679.8A CN116737088B (en) | 2023-08-15 | 2023-08-15 | Object migration method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311021679.8A CN116737088B (en) | 2023-08-15 | 2023-08-15 | Object migration method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116737088A CN116737088A (en) | 2023-09-12 |
| CN116737088B true CN116737088B (en) | 2023-11-03 |
Family
ID=87911842
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311021679.8A Active CN116737088B (en) | 2023-08-15 | 2023-08-15 | Object migration method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116737088B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107037978A (en) * | 2016-10-31 | 2017-08-11 | 福建亿榕信息技术有限公司 | Data Migration bearing calibration and system |
| CN110377580A (en) * | 2019-07-25 | 2019-10-25 | 中国工商银行股份有限公司 | A kind of data migration method, device and equipment |
| CN111984395A (en) * | 2019-05-22 | 2020-11-24 | 中移(苏州)软件技术有限公司 | Data migration method and system, and computer readable storage medium |
| CN113282246A (en) * | 2021-06-15 | 2021-08-20 | 杭州海康威视数字技术股份有限公司 | Data processing method and device |
| CN114416691A (en) * | 2021-12-27 | 2022-04-29 | 北京像素软件科技股份有限公司 | Data migration method and device, electronic equipment and readable storage medium |
-
2023
- 2023-08-15 CN CN202311021679.8A patent/CN116737088B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107037978A (en) * | 2016-10-31 | 2017-08-11 | 福建亿榕信息技术有限公司 | Data Migration bearing calibration and system |
| CN111984395A (en) * | 2019-05-22 | 2020-11-24 | 中移(苏州)软件技术有限公司 | Data migration method and system, and computer readable storage medium |
| CN110377580A (en) * | 2019-07-25 | 2019-10-25 | 中国工商银行股份有限公司 | A kind of data migration method, device and equipment |
| CN113282246A (en) * | 2021-06-15 | 2021-08-20 | 杭州海康威视数字技术股份有限公司 | Data processing method and device |
| CN114416691A (en) * | 2021-12-27 | 2022-04-29 | 北京像素软件科技股份有限公司 | Data migration method and device, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116737088A (en) | 2023-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10114682B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| US20190213085A1 (en) | Implementing Fault Domain And Latency Requirements In A Virtualized Distributed Storage System | |
| US9998531B2 (en) | Computer-based, balanced provisioning and optimization of data transfer resources for products and services | |
| US9569291B1 (en) | Systems and methods for inter-process messaging | |
| US9092365B2 (en) | Splitting direct memory access windows | |
| CN110308984B (en) | Cross-cluster computing system for processing geographically distributed data | |
| CN111324427B (en) | Task scheduling method and device based on DSP | |
| CN110119304B (en) | Interrupt processing method and device and server | |
| WO2021114025A1 (en) | Incremental data determination method, incremental data determination apparatus, server and terminal device | |
| CN111966289A (en) | Partition optimization method and system based on Kafka cluster | |
| CN104317716B (en) | Data transmission method and distributed node equipment between distributed node | |
| CN103236989A (en) | Cache control method, devices and system in content delivery network | |
| GB2555682A (en) | Repartitioning data in a distributed computing system | |
| US9104600B2 (en) | Merging direct memory access windows | |
| CN115827250A (en) | Data storage method, device and equipment | |
| CN111400241B (en) | Data reconstruction method and device | |
| US20110302377A1 (en) | Automatic Reallocation of Structured External Storage Structures | |
| CN116737088B (en) | Object migration method and device, electronic equipment and storage medium | |
| CN110557341A (en) | Method and device for limiting data current | |
| US20160085683A1 (en) | Data receiving device and data receiving method | |
| CN109951506B (en) | Method and equipment for evaluating performance of storage cluster | |
| US10824640B1 (en) | Framework for scheduling concurrent replication cycles | |
| US20190245799A1 (en) | Reliability processing of remote direct memory access | |
| WO2022021357A1 (en) | File block download method and apparatus | |
| CN112988383A (en) | Resource allocation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |