CN114936003B - Data layered migration method, device and equipment of resource pool and readable storage medium - Google Patents
Data layered migration method, device and equipment of resource pool and readable storage medium Download PDFInfo
- Publication number
- CN114936003B CN114936003B CN202210483991.8A CN202210483991A CN114936003B CN 114936003 B CN114936003 B CN 114936003B CN 202210483991 A CN202210483991 A CN 202210483991A CN 114936003 B CN114936003 B CN 114936003B
- Authority
- CN
- China
- Prior art keywords
- resource pool
- information
- data
- hot spot
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of computers, in particular to a data layered migration method, a device, equipment and a readable storage medium of a resource pool, wherein the method comprises the steps of acquiring data information input by a user, structure information of the resource pool and capacity information of the resource pool; dividing the structural information of the resource pool into a hot spot resource pool and at least one non-hot spot resource pool; calculating a data migration threshold of the hot spot resource pool according to the data information input by the user; and judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the capacity information of the resource pool to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result. According to the method and the device, the data in the resource pool are migrated to other types of resource pools according to the set rule, so that the original resource pool has more space to continuously write the data, the vacated space can continuously receive the user data, and good read-write performance is ensured.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data layered migration method, a data layered migration device, data layered migration equipment and a readable storage medium for a resource pool.
Background
In a CEPH object gateway, a current data space capacity expansion method is still immature, when an object gateway is created, only one resource pool for storing data is created by default, under the condition, data reading and writing are all concentrated in the data resource pool, conflicts between performance and available capacity occur, if good performance is expected, a solid state disk resource pool needs to be adopted, so that the capacity of the available resource pool is insufficient, and if high capacity is expected, a traditional hard disk resource pool needs to be adopted, so that compromise needs to be made on performance, and a hard disk resource pool expansion method with good performance and high capacity is needed at present.
Disclosure of Invention
The present invention provides a method, an apparatus, a device and a readable storage medium for layered migration of data in a resource pool, so as to improve the above problems. In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
in one aspect, the present application provides a method for data layered migration of a resource pool, where the method includes: acquiring first information, second information and third information, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, a read-write mode and thread number input by the user; sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool, wherein the layering module is a module for layering the resource pool structure; sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, wherein the computing module is a module for computing the data migration threshold of the hotspot resource pool; and judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result.
In a second aspect, an embodiment of the present application provides an apparatus for data layered migration of a resource pool, where the apparatus includes: the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, reading and writing modes and thread number input by the user; the first sending unit is used for sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool, and the layering module is a module for layering the resource pool structure; the first computing unit is used for sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, and the computing module is a module for computing the data migration threshold of the hotspot resource pool; and the first judging unit is used for judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result.
In a third aspect, an embodiment of the present application provides a device for data hierarchical migration of a resource pool, where the device includes a memory and a processor. The memory is used for storing a computer program; the processor is used for realizing the steps of the data hierarchical migration method of the resource pool when executing the computer program.
In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the data hierarchical migration method for a resource pool are implemented.
The beneficial effects of the invention are as follows:
according to the method and the system, the attribute of the data resource pool of the object gateway is modified, so that the type of the user-defined resource pool can be added in the object gateway, the resource pool of the type is created, the type of the resource pool to be written can be selected when data is written, and the data of the resource pool is transferred to the resource pools of other types according to the set rule, so that the original resource pool has more space to continuously write the data.
The invention also creates the hot spot resource pool and the non-hot spot resource pool, the space of the non-hot spot resource pool is large enough to contain more data, and the vacated space can continue to receive the user data after the data of the hot spot resource pool is moved away, thereby ensuring good read-write performance.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic flow chart of a data layered migration method for a resource pool according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data hierarchical migration apparatus of a resource pool according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data hierarchical migration device of a resource pool according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not construed as indicating or implying relative importance.
Example 1
As shown in fig. 1, the present embodiment provides a method for hierarchically migrating data of a resource pool, where the method includes step S1, step S2, step S3, and step S4.
S1, acquiring first information, second information and third information, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, reading and writing modes and thread number input by the user;
s2, sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool, wherein the layering module is a module for layering the resource pool structure;
s3, sending the first information to a computing module to obtain a data migration threshold of the hot spot resource pool, wherein the computing module is a module for computing the data migration threshold of the hot spot resource pool;
and S4, judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result.
It can be understood that the invention can add the self-defined resource pool type in the object gateway by modifying the attribute of the object gateway data resource pool, and create the resource pool of the type, then can select the written resource pool type when writing data, and migrate the data of the resource pool to other types of resource pools according to the set rule, so that the original resource pool has more space to continue writing data.
The invention also creates the hot spot resource pool and the non-hot spot resource pool, the space of the non-hot spot resource pool is large enough to contain more data, and the vacated space can continue to receive the user data after the data of the hot spot resource pool is moved away, thereby ensuring good read-write performance.
In a specific embodiment of the present disclosure, the step S2 includes a step S21, a step S22, and a step S234.
Step S21, calling preset hard disk equipment information;
step S22, mapping the structural information of the resource pool based on the preset hard disk device information, wherein the solid state disk information is mapped to a first sub-resource pool in the resource pool, and the traditional hard disk information is mapped to a second sub-resource pool in the resource pool, wherein the first sub-resource pool is a resource pool for receiving hot spot data, and the second sub-resource pool is a resource pool for receiving migration data of the first sub-resource pool;
step S23, modifying the structures of the first sub-resource pool and the second sub-resource pool based on a preset resource pool type, wherein an attribute type of the first sub-resource pool is determined, an attribute type of at least one second sub-resource pool is determined, and the structures of the corresponding first sub-resource pool and the corresponding second sub-resource pool are modified based on the determined attribute types, so as to obtain a hot spot resource pool and at least one non-hot spot resource pool.
The resource pools are classified according to different resource pool attributes, wherein a common resource pool is used as a hot spot resource pool, the resource pool which is established by a solid state disk is used as the hot spot resource pool, the resource pool which is established by a traditional hard disk is used as a non-hot spot resource pool, the hot spot resource pool is used for storing common data, the non-hot spot resource pool is used for storing common data, and the resource pools are layered to obtain the multi-level resource pools.
In a specific embodiment of the present disclosure, the step S3 includes a step S31, a step S32, and a step S33.
Step S31, calling data information input by a user in the first information;
step S32, creating a plurality of execution threads based on the data information input by the user, wherein the execution threads are correspondingly created according to the thread number in the data information input by the user, performing read-write operation on the read-write mode, recording the file size information of each read-write operation and the completion time of each read-write operation in the read-write process, and determining the read-write performance value of the storage device under the condition of the utilization rate of the hotspot resource pool according to a preset read-write performance table;
and S33, clustering the read-write performance values of the storage equipment under the condition of the utilization rate of the hot spot resource pool based on a distance clustering algorithm to obtain at least one read-write performance value cluster, wherein the distance from each cluster point to a cluster central point is calculated based on an Euclidean distance calculation formula, the average value of the distances is calculated, and the average value is used as a data migration threshold value of the hot spot resource pool.
It can be understood that the invention operates by establishing a plurality of threads, and then judges the read-write performance of each thread, and further uses the read-write performance as the data migration threshold of the hot spot resource pool, and determines whether the layered resource pool is in an overload operation state, if the layered resource pool is in the overload operation state, the data in the hot spot resource pool is immediately migrated into the non-hot spot resource pool, and the space in the hot spot resource pool is expanded while the system is ensured to normally operate.
In a specific embodiment of the present disclosure, the step S4 includes a step S41, a step S42, a step S43, and a step S44.
S41, calling a data migration threshold of the hot spot resource pool;
step S42, calculating the ratio of the first information to the third information based on a capacity utilization rate formula of a resource pool to obtain the utilization rate of the hotspot resource pool;
step S43, comparing the utilization rate of the hot spot resource pool with the data migration threshold of the resource pool, and judging whether the utilization rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool to obtain judgment result information.
It can be understood that the method and the device compare the capacity utilization rate of the hot spot resource pool with the data migration threshold of the resource pool based on the capacity utilization rate of the hot spot resource pool, so that the capacity utilization rate of the hot spot resource pool is utilized to the maximum, the resource waste is reduced, and the performance reduction caused by excessive data in the hot spot resource pool is prevented.
In a specific embodiment of the present disclosure, the step S4 includes a step S44 and a step S45.
Step S44, if the utilization rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, sending a first sub-command, wherein the first sub-command is used for migrating data in the hot spot resource pool into preset hard disk equipment for storage;
step S45, if the utilization rate of the hot spot resource pool is smaller than the data migration threshold of the resource pool, sending a second subcommand, comparing the utilization rate of the hot spot resource pool with the data migration threshold of the hot spot resource pool at an interval of 30S until the utilization rate of the hot spot resource pool is larger than or equal to the data migration threshold of the hot spot resource pool, and migrating the data in the hot spot resource pool to a preset hard disk device for storage.
It can be understood that the invention controls the migration of data in the resource pool by sending a command, and checks whether the data in the hot spot resource pool reaches the roe every 30S, so as to migrate the data, wherein the 30S time can be adjusted at any time, so as to ensure that the performance state of the resource pool is in the optimal state at any time and any place, thereby ensuring the stable operation of the system, and the non-hot spot data can be stored by migrating the data to the non-resource pool, thereby preventing the loss of the data and causing the loss of the data.
In a specific embodiment of the present disclosure, step S4 is followed by step S5, step S6, step S7, and step S8.
S5, calling record information of historical read-write operation;
s6, marking abnormal records in the record information of the historical read-write operation to obtain marked record information;
s7, obtaining a CART decision tree based on a CART algorithm and the marked recording information, carrying out random pruning treatment on the CART decision tree, determining a constant of the CART decision tree, optimizing the decision tree based on the constant of the CART decision tree and a Gini index calculation method to obtain an optimal decision tree, sending the read-write operation of the storage equipment to the optimal decision tree to judge whether the read-write operation is abnormal or not, and obtaining abnormal data of the read-write operation of the storage equipment;
and S8, storing the abnormal data of the read-write operation of the storage equipment to an abnormal resource pool of a non-hotspot resource pool.
It can be understood that the invention obtains a judgment model of whether the trained data is abnormal or not by marking the historical abnormal information and then training through the CART algorithm, judges the data in the storage device based on the judgment model, and then directly puts the abnormal data into the non-hot resource pool, thereby preventing the data in the hot resource pool from being lost or damaged, and also checking and clearing the abnormal data by sending alarm information to the communication device of the manager.
Example 2
As shown in fig. 2, the present embodiment provides a device for migrating data of a resource pool in a hierarchical manner, where the device includes a first obtaining unit 701, a first sending unit 702, a first calculating unit 703, and a first determining unit 704.
A first obtaining unit 701, configured to obtain first information, second information, and third information, where the first information includes data information input by a user, the second information is structure information of a resource pool, the third information is capacity information of the resource pool, and the data information includes file information, a file size, a read-write mode, and a thread count input by the user;
a first sending unit 702, configured to send the second information to a layering module, so as to obtain a hot spot resource pool and at least one non-hot spot resource pool, where the layering module is a module that performs layering on a resource pool structure;
the first calculating unit 703 is configured to send the first information to a calculating module to obtain a data migration threshold of the hotspot resource pool, where the calculating module is a module for calculating the data migration threshold of the hotspot resource pool;
a first determining unit 704, configured to determine, based on the first information and the third information, whether a usage rate of a hot spot resource pool is greater than a data migration threshold of the hot spot resource pool, to obtain a determination result, and send a command of data migration of the hot spot resource pool based on the determination result.
In a specific embodiment of the present disclosure, the first sending unit 702 includes a first invoking subunit 7021, a first processing subunit 7022, and a second processing subunit 7023.
A first calling subunit 7021, configured to call preset hard disk device information;
a first processing subunit 7022, configured to map structure information of the resource pool based on the preset hard disk device information, where solid state disk information is mapped to a first sub-resource pool in the resource pool, and traditional hard disk information is mapped to a second sub-resource pool in the resource pool, where the first sub-resource pool is a resource pool that receives hot spot data, and the second sub-resource pool is a resource pool that receives migration data of the first sub-resource pool;
a second processing subunit 7023, configured to modify the structures of the first sub-resource pool and the second sub-resource pool based on a preset resource pool type, where an attribute type of the first sub-resource pool is determined, an attribute type of at least one second sub-resource pool is determined, and the corresponding structures of the first sub-resource pool and the at least one second sub-resource pool are modified based on the determined attribute types, so as to obtain a hot resource pool and at least one non-hot resource pool.
In a specific embodiment of the present disclosure, the first calculating unit 703 includes a second invoking subunit 7031, a third processing subunit 7032, and a fourth processing subunit 7033.
A second calling subunit 7031, configured to call data information input by the user in the first information;
a third processing subunit 7032, configured to create multiple execution threads based on the data information input by the user, where the execution threads are created according to the number of threads in the data information input by the user, perform read-write operation on the read-write manner, record file size information of each read-write operation and completion time of each read-write operation in the read-write process, and determine a read-write performance value of the storage device under the condition of the usage rate of the hotspot resource pool according to a preset read-write performance table;
a fourth processing subunit 7033, configured to perform clustering processing on the read-write performance values of the storage device under the condition of the usage rate of the hot spot resource pool based on a distance-based clustering algorithm to obtain at least one read-write performance value cluster, where a distance from a point of each cluster to a cluster center point is obtained by calculation based on an euclidean distance calculation formula, an average value of the distances is calculated, and the average value is used as a data migration threshold of the hot spot resource pool.
In a specific embodiment of the present disclosure, the first determining unit 704 includes a third invoking sub-unit 7041, a fifth processing sub-unit 7042, and a sixth processing sub-unit 7043.
A third invoking subunit 7041, configured to invoke a data migration threshold of the hotspot resource pool;
a fifth processing subunit 7042, configured to calculate a ratio of the first information to the third information based on a capacity utilization formula of a resource pool, to obtain a utilization of the hotspot resource pool;
a sixth processing subunit 7043, configured to compare the usage rate of the hot spot resource pool with the data migration threshold of the resource pool, determine whether the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, and obtain determination result information.
In a specific embodiment of the present disclosure, the first determining unit 704 further includes a first determining subunit 7044 and a second determining subunit 7045.
A first determining subunit 7044, configured to send a first sub-command if the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, where the first sub-command is to migrate data in the hot spot resource pool to a preset hard disk device for storage;
a second determining subunit 7045, configured to send a second subcommand if the usage rate of the hot spot resource pool is less than the data migration threshold of the resource pool, where the second subcommand compares the usage rate of the hot spot resource pool with the data migration threshold of the hot spot resource pool at an interval of 30S until the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, and migrates the data in the hot spot resource pool into a preset hard disk device for storage.
In a specific embodiment of the present disclosure, the first determining unit 704 includes a first calling unit 705, a first processing unit 706, a second processing unit 707, and a third processing unit 708.
A first calling unit 705, configured to call record information of a history read-write operation;
the first processing unit 706 is configured to mark an abnormal record in the record information of the historical read-write operation, so as to obtain marked record information;
a second processing unit 707, configured to obtain a CART decision tree based on a CART algorithm and the marked recording information, perform random pruning on the CART decision tree, determine a constant of the CART decision tree, optimize the decision tree based on the constant of the CART decision tree and a kini index calculation method, obtain an optimal decision tree, send a read-write operation of a storage device to the optimal decision tree, determine whether the read-write operation is abnormal, and obtain abnormal data of the read-write operation of the storage device;
the third processing unit 708 is configured to store the abnormal data of the read-write operation of the storage device to an abnormal resource pool of the non-hotspot resource pool.
It should be noted that, regarding the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated herein.
Example 3
Corresponding to the above method embodiment, the embodiment of the present disclosure further provides a data layered migration device of a resource pool, where the data layered migration device of a resource pool described below and the data layered migration method of a resource pool described above may be referred to correspondingly.
FIG. 3 is a block diagram illustrating a data hierarchical migration apparatus 800 of a resource pool in accordance with an illustrative embodiment. As shown in fig. 3, the data hierarchical migration apparatus 800 of the resource pool may include: a processor 801, a memory 802. The data layered migration apparatus 800 of the resource pool may further include one or more of a multimedia component 803, an input/output (I/O) interface 804, and a communication component 805.
The processor 801 is configured to control the overall operation of the data hierarchical migration apparatus 800 of the resource pool, so as to complete all or part of the steps in the data hierarchical migration method of the resource pool. The memory 802 is used to store various types of data to support the operation of the data hierarchical migration device 800 at the resource pool, which may include, for example, instructions for any application or method operating on the data hierarchical migration device 800 at the resource pool, as well as application-related data, such as contact data, transceived messages, pictures, audio, video, and so forth. The Memory 802 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically Erasable Programmable Read-Only Memory (EEPROM), erasable Programmable Read-Only Memory (EPROM), programmable Read-Only Memory (PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. The multimedia components 803 may include screen and audio components. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may further be stored in the memory 802 or transmitted through the communication component 805. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 804 provides an interface between the processor 801 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the data hierarchical migration device 800 and other devices of the resource pool. Wireless communication, such as Wi-Fi, bluetooth, near Field Communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them, so that the corresponding communication component 805 may include: wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the data hierarchical migration apparatus 800 of the resource pool may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, and is used for executing one of the above-mentioned data hierarchical migration methods of the resource pool.
In another exemplary embodiment, a computer readable storage medium is also provided, which comprises program instructions, which when executed by a processor, implement the steps of the above-described method for data hierarchical migration of a resource pool. For example, the computer readable storage medium may be the above-mentioned memory 802 comprising program instructions executable by the processor 801 of the data hierarchical migration apparatus 800 of the resource pool to perform the above-mentioned data hierarchical migration method of the resource pool.
Example 4
Corresponding to the above method embodiment, the present disclosure further provides a readable storage medium, and a readable storage medium described below and a data hierarchical migration method of a resource pool described above may be referred to correspondingly.
A readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the data hierarchical migration method of a resource pool of the above-mentioned method embodiments.
The readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various other readable storage media capable of storing program codes.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A method for layered migration of data in a resource pool is characterized by comprising the following steps:
acquiring first information, second information and third information, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, reading and writing modes and thread number input by the user;
sending the second information to a layering module to obtain a hot resource pool and at least one non-hot resource pool, wherein the layering module is used for layering the resource pool structure;
sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, wherein the computing module is a module for computing the data migration threshold of the hotspot resource pool;
judging whether the utilization rate of the hot spot resource pool is greater than a data migration threshold value of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result;
sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, wherein the method comprises the following steps:
calling data information input by a user in the first information;
creating a plurality of execution threads based on the data information input by the user, wherein the execution threads are correspondingly created according to the thread number in the data information input by the user, performing read-write operation on the read-write mode, recording the file size information of each read-write operation and the completion time of each read-write operation in the read-write process, and determining the read-write performance value of the storage device under the condition of the utilization rate of the hotspot resource pool according to a preset read-write performance table;
clustering the read-write performance values of the storage device under the condition of the utilization rate of the hot spot resource pool based on a distance clustering algorithm to obtain at least one read-write performance value cluster, wherein the distance from a point of each cluster to a cluster central point is calculated based on an Euclidean distance calculation formula, the average value of the distances is calculated, and the average value is used as a data migration threshold value of the hot spot resource pool.
2. The method for the layered migration of data in a resource pool according to claim 1, wherein the step of sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool comprises:
calling preset hard disk equipment information;
mapping the structural information of the resource pool based on the preset hard disk device information, wherein the solid state disk information is mapped to a first sub-resource pool in the resource pool, and the traditional hard disk information is mapped to a second sub-resource pool in the resource pool, wherein the first sub-resource pool is a resource pool for receiving hot spot data, and the second sub-resource pool is a resource pool for receiving migration data of the first sub-resource pool;
and modifying the structures of the first sub resource pool and the second sub resource pool based on a preset resource pool type, wherein the attribute type of the first sub resource pool is determined, the attribute type of at least one second sub resource pool is determined, and the corresponding structures of the first sub resource pool and the at least one second sub resource pool are modified based on the determined attribute types to obtain a hot resource pool and at least one non-hot resource pool.
3. The method for layered data migration of a resource pool according to claim 1, wherein determining whether the usage rate of a hot spot resource pool is greater than a data migration threshold of the hot spot resource pool based on the first information and the third information includes:
calling a data migration threshold of the hot spot resource pool;
calculating the ratio of the first information to the third information based on a capacity utilization rate formula of a resource pool to obtain the utilization rate of the hotspot resource pool;
and comparing the utilization rate of the hot spot resource pool with the data migration threshold of the resource pool, and judging whether the utilization rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool to obtain judgment result information.
4. An apparatus for layered migration of data from a resource pool, comprising:
the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, reading and writing modes and thread number input by the user;
the first sending unit is used for sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool, and the layering module is a module for layering the resource pool structure;
the first computing unit is used for sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, and the computing module is a module for computing the data migration threshold of the hotspot resource pool;
the first judging unit is used for judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result;
wherein the first calculation unit includes:
the second calling subunit is used for calling the data information input by the user in the first information;
a third processing subunit, configured to create multiple execution threads based on the data information input by the user, where an execution thread is created according to a thread number in the data information input by the user, perform read-write operation on the read-write mode, record file size information of each read-write operation and completion time of each read-write operation in a read-write process, and determine a read-write performance value of the storage device under the condition of the usage rate of the hotspot resource pool according to a preset read-write performance table;
and the fourth processing subunit is used for clustering the read-write performance values of the storage device under the condition of the utilization rate of the hot spot resource pool based on a distance clustering algorithm to obtain at least one read-write performance value cluster, wherein the distance from each cluster point to a cluster central point is calculated based on an Euclidean distance calculation formula, the average value of the distances is calculated, and the average value is used as the data migration threshold value of the hot spot resource pool.
5. The apparatus for data hierarchical migration of resource pool according to claim 4, wherein said apparatus comprises:
the first calling subunit is used for calling preset hard disk equipment information;
the first processing subunit is configured to map structural information of the resource pool based on the preset hard disk device information, wherein the solid state disk information is mapped to a first sub-resource pool in the resource pool, and the conventional hard disk information is mapped to a second sub-resource pool in the resource pool, the first sub-resource pool is a resource pool for receiving hot spot data, and the second sub-resource pool is a resource pool for receiving migration data of the first sub-resource pool;
and the second processing subunit is configured to modify the structures of the first sub-resource pool and the second sub-resource pool based on a preset resource pool type, determine an attribute type of the first sub-resource pool and determine an attribute type of at least one second sub-resource pool, and modify the structures of the corresponding first sub-resource pool and the corresponding second sub-resource pool based on the determined attribute types to obtain a hot spot resource pool and at least one non-hot spot resource pool.
6. The apparatus for layered data migration of a resource pool according to claim 4, wherein the determining whether the usage rate of the hot spot resource pool is greater than the data migration threshold of the hot spot resource pool based on the first information and the third information includes:
the third calling subunit is used for calling the data migration threshold of the hotspot resource pool;
the fifth processing subunit is configured to calculate a ratio of the first information to the third information based on a capacity utilization formula of a resource pool, so as to obtain a utilization rate of the hotspot resource pool;
and the sixth processing subunit is configured to compare the usage rate of the hot spot resource pool with the data migration threshold of the resource pool, determine whether the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, and obtain determination result information.
7. A device for layered migration of data from a resource pool, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data hierarchical migration method of the resource pool according to any one of claims 1 to 3 when executing the computer program.
8. A readable storage medium, characterized by: the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of a method for hierarchical migration of data of a resource pool according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210483991.8A CN114936003B (en) | 2022-05-06 | 2022-05-06 | Data layered migration method, device and equipment of resource pool and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210483991.8A CN114936003B (en) | 2022-05-06 | 2022-05-06 | Data layered migration method, device and equipment of resource pool and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114936003A CN114936003A (en) | 2022-08-23 |
| CN114936003B true CN114936003B (en) | 2023-03-21 |
Family
ID=82865243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210483991.8A Active CN114936003B (en) | 2022-05-06 | 2022-05-06 | Data layered migration method, device and equipment of resource pool and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114936003B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110688067A (en) * | 2019-09-06 | 2020-01-14 | 苏州浪潮智能科技有限公司 | Layering method, device, equipment and medium of storage system |
| CN113836116A (en) * | 2021-09-29 | 2021-12-24 | 济南浪潮数据技术有限公司 | Data migration method and device, electronic equipment and readable storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294440B (en) * | 2015-05-27 | 2019-06-07 | 阿里巴巴集团控股有限公司 | The method and apparatus of data real-time migration |
| CN106502576B (en) * | 2015-09-06 | 2020-06-23 | 中兴通讯股份有限公司 | Migration strategy adjusting method and device |
| CN106502578B (en) * | 2015-09-06 | 2019-06-11 | 中兴通讯股份有限公司 | Capacity changes suggesting method and device |
-
2022
- 2022-05-06 CN CN202210483991.8A patent/CN114936003B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110688067A (en) * | 2019-09-06 | 2020-01-14 | 苏州浪潮智能科技有限公司 | Layering method, device, equipment and medium of storage system |
| CN113836116A (en) * | 2021-09-29 | 2021-12-24 | 济南浪潮数据技术有限公司 | Data migration method and device, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114936003A (en) | 2022-08-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105740078B (en) | Memory management method, device and terminal | |
| WO2022142551A1 (en) | Form processing method and apparatus, and medium and computer device | |
| CN109614045B (en) | Metadata dropping method and device and related equipment | |
| CN105989015B (en) | Database capacity expansion method and device and method and device for accessing database | |
| CN104426944A (en) | Method and device for information feedback, and terminal | |
| CN111274242A (en) | Data searching method and device of tree structure for hospital logistics operation and maintenance | |
| CN114328281A (en) | Method and device for processing abnormal power failure of solid state disk, electronic equipment and medium | |
| US9785374B2 (en) | Storage device management in computing systems | |
| CN114936003B (en) | Data layered migration method, device and equipment of resource pool and readable storage medium | |
| CN110442636B (en) | Data reading and writing method and device and data reading and writing equipment | |
| CN110347546B (en) | Dynamic adjustment method, device, medium and electronic equipment for monitoring task | |
| CN108399118B (en) | File system test data processing method and device, storage medium and terminal | |
| CN115408546A (en) | Time sequence data management method, device, equipment and storage medium | |
| CN111459848B (en) | File fragment sorting method and device, storage medium and electronic equipment | |
| CN114579136A (en) | Code processing method and device, computer equipment and storage medium | |
| CN115185458A (en) | Data writing method and device, storage medium and computer equipment | |
| CN107623784A (en) | Method for information display and device, terminal installation and computer-readable recording medium | |
| CN114186976A (en) | Workflow transfer method and device, computer equipment and storage medium | |
| CN114201729A (en) | Method, device and equipment for selecting matrix operation mode and storage medium | |
| WO2020210975A1 (en) | Sub-trajectory selection method, electronic device, and computer readable storage medium | |
| CN114489519B (en) | IO request processing method, device, equipment and readable storage medium | |
| CN114879916B (en) | Method and device for managing storage unit | |
| US20230065549A1 (en) | Dynamically adaptable virtual keyboard | |
| CN112764828B (en) | Business logic management method and device, computer equipment and medium | |
| CN117390462A (en) | User behavior analysis method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |