CN114936003A - Data layered migration method, device and equipment of resource pool and readable storage medium - Google Patents
Data layered migration method, device and equipment of resource pool and readable storage medium Download PDFInfo
- Publication number
- CN114936003A CN114936003A CN202210483991.8A CN202210483991A CN114936003A CN 114936003 A CN114936003 A CN 114936003A CN 202210483991 A CN202210483991 A CN 202210483991A CN 114936003 A CN114936003 A CN 114936003A
- Authority
- CN
- China
- Prior art keywords
- resource pool
- information
- data
- hot spot
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of computers, in particular to a data layered migration method, a device, equipment and a readable storage medium of a resource pool, wherein the method comprises the steps of acquiring data information input by a user, structure information of the resource pool and capacity information of the resource pool; dividing the structural information of the resource pool into a hot spot resource pool and at least one non-hot spot resource pool; calculating a data migration threshold of the hot resource pool according to the data information input by the user; and judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the capacity information of the resource pool to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result. According to the method and the device, the data in the resource pool are migrated to other types of resource pools according to the set rule, so that the original resource pool has more space to continuously write the data, the vacated space can continuously receive the user data, and good read-write performance is ensured.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method, a device and equipment for layered migration of data of a resource pool and a readable storage medium.
Background
In a CEPH object gateway, a current data space capacity expansion method is not mature, when the object gateway is established, only one resource pool for storing data is established by default, under the condition, data reading and writing are concentrated in the data resource pool, conflicts between performance and available capacity occur, if good performance is expected, a solid state disk resource pool is adopted, the capacity of the available resource pool is insufficient, and if large capacity is expected, a traditional hard disk resource pool is adopted, so that compromise on performance is required, and a hard disk resource pool capacity expansion method with good performance and large capacity is required at present.
Disclosure of Invention
The invention aims to provide a method, a device and equipment for layered migration of data of a resource pool and a readable storage medium, so as to solve the problems. In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
in one aspect, the present application provides a method for data layered migration of a resource pool, where the method includes: acquiring first information, second information and third information, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, a read-write mode and thread number input by the user; sending the second information to a layering module to obtain a hot resource pool and at least one non-hot resource pool, wherein the layering module is used for layering the resource pool structure; sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, wherein the computing module is a module for computing the data migration threshold of the hotspot resource pool; and judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a hot spot resource pool data migration command based on the judgment result.
In a second aspect, an embodiment of the present application provides an apparatus for data layered migration of a resource pool, where the apparatus includes: the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, reading and writing modes and thread number input by the user; the first sending unit is used for sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool, and the layering module is a module for layering the resource pool structure; the first computing unit is used for sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, and the computing module is a module for computing the data migration threshold of the hotspot resource pool; and the first judging unit is used for judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result.
In a third aspect, an embodiment of the present application provides a data hierarchical migration device for a resource pool, where the device includes a memory and a processor. The memory is used for storing a computer program; the processor is used for realizing the steps of the data hierarchical migration method of the resource pool when executing the computer program.
In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps of the data hierarchical migration method for a resource pool.
The invention has the beneficial effects that:
according to the method and the system, the attribute of the data resource pool of the object gateway is modified, so that the type of the user-defined resource pool can be added in the object gateway, the resource pool of the type is created, the type of the resource pool to be written can be selected when data is written, and the data of the resource pool is transferred to the resource pools of other types according to the set rule, so that the original resource pool has more space to continuously write the data.
The invention also creates the hot spot resource pool and the non-hot spot resource pool, the space of the non-hot spot resource pool is large enough to contain more data, and the vacated space can continue to receive the user data after the data of the hot spot resource pool is moved away, thereby ensuring good read-write performance.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a schematic flow chart of a data layered migration method for a resource pool according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data hierarchical migration apparatus of a resource pool according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data hierarchical migration device of a resource pool according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not construed as indicating or implying relative importance.
Example 1
As shown in fig. 1, the present embodiment provides a data hierarchical migration method of a resource pool, where the method includes step S1, step S2, step S3, and step S4.
Step S1, acquiring first information, second information and third information, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, reading and writing modes and thread number input by the user;
step S2, sending the second information to a layering module to obtain a hot resource pool and at least one non-hot resource pool, wherein the layering module is a module for layering the resource pool structure;
step S3, sending the first information to a computing module to obtain a data migration threshold of the hot spot resource pool, wherein the computing module is a module for computing the data migration threshold of the hot spot resource pool;
step S4, determining whether the usage rate of the hot resource pool is greater than the data migration threshold of the hot resource pool based on the first information and the third information, obtaining a determination result, and sending a command for data migration of the hot resource pool based on the determination result.
It can be understood that the invention can add the self-defined resource pool type in the object gateway by modifying the attribute of the object gateway data resource pool, and create the resource pool of the type, then can select the written resource pool type when writing data, and migrate the data of the resource pool to other types of resource pools according to the set rule, so that the original resource pool has more space to continue writing data.
The invention also creates the hot spot resource pool and the non-hot spot resource pool, the space of the non-hot spot resource pool is large enough to contain more data, and the vacated space can continue to receive the user data after the data of the hot spot resource pool is moved away, thereby ensuring good read-write performance.
In a specific embodiment of the present disclosure, the step S2 includes a step S21, a step S22 and a step S234.
Step S21, calling preset hard disk device information;
step S22, mapping the structural information of the resource pool based on the preset hard disk device information, wherein the solid state disk information is mapped to a first sub-resource pool in the resource pool, and the traditional hard disk information is mapped to a second sub-resource pool in the resource pool, the first sub-resource pool is a resource pool for receiving hot data, and the second sub-resource pool is a resource pool for receiving migration data of the first sub-resource pool;
step S23, modifying the structures of the first sub resource pool and the second sub resource pool based on a preset resource pool type, wherein an attribute type of the first sub resource pool and an attribute type of at least one of the second sub resource pools are determined, and modifying the structures of the corresponding first sub resource pool and the corresponding second sub resource pool based on the determined attribute types to obtain a hot resource pool and at least one non-hot resource pool.
It can be understood that the resource pools are classified according to different resource pool attributes by establishing the first sub-resource pool and the second sub-resource pool and modifying the attribute of each resource pool, wherein the commonly used resource pool is used as a hot spot resource pool, the resource pool which is established by the solid state disk is required to be used as the hot spot resource pool, the resource pool established by the traditional hard disk is used as a non-hot spot resource pool, the hot spot resource pool is used for storing commonly used data, the non-hot spot resource pool is used for storing general data, and the resource pools are layered to obtain the multi-level resource pool.
In a specific embodiment of the present disclosure, the step S3 includes steps S31, S32 and S33.
Step S31, calling the data information input by the user in the first information;
step S32, creating a plurality of execution threads based on the data information input by the user, wherein the execution threads are correspondingly created according to the thread number in the data information input by the user, performing read-write operation on the read-write mode, recording the file size information of each read-write operation and the completion time of each read-write operation in the read-write process, and determining the read-write performance value of the storage device under the condition of the utilization rate of the hot resource pool according to a preset read-write performance table;
step S33, clustering the read-write performance values of the storage device under the condition of the utilization rate of the hot spot resource pool based on a distance clustering algorithm to obtain at least one read-write performance value cluster, wherein the distance from each cluster point to a cluster central point is calculated based on an Euclidean distance calculation formula, the average value of the distances is calculated, and the average value is used as the data migration threshold value of the hot spot resource pool.
It can be understood that the invention operates by establishing a plurality of threads, and then judges the read-write performance of each thread, and further uses the read-write performance as the data migration threshold of the hot spot resource pool, and determines whether the layered resource pool is in an overload operation state, if the layered resource pool is in the overload operation state, the data in the hot spot resource pool is immediately migrated into the non-hot spot resource pool, and the space in the hot spot resource pool is expanded while the system is ensured to normally operate.
In a specific embodiment of the present disclosure, the step S4 includes steps S41, S42, S43 and S44.
Step S41, calling a data migration threshold of the hot spot resource pool;
step S42, calculating the ratio of the first information to the third information based on a capacity utilization rate formula of a resource pool to obtain the utilization rate of the hot spot resource pool;
step S43, comparing the usage rate of the hot spot resource pool with the data migration threshold of the resource pool, and determining whether the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, so as to obtain determination result information.
It can be understood that the method and the device compare the capacity utilization rate of the hot spot resource pool with the data migration threshold of the resource pool based on the capacity utilization rate of the hot spot resource pool, so that the capacity utilization rate of the hot spot resource pool is utilized to the maximum, the resource waste is reduced, and the performance reduction caused by excessive data in the hot spot resource pool is prevented.
In a specific embodiment of the present disclosure, the step S4 includes steps S44 and S45.
Step S44, if the utilization rate of the hot spot resource pool is greater than or equal to the data migration threshold value of the hot spot resource pool, sending a first sub-command, wherein the first sub-command is to migrate data in the hot spot resource pool into a preset hard disk device for storage;
step S45, if the usage rate of the hot spot resource pool is less than the data migration threshold of the resource pool, sending a second subcommand, which is to compare the usage rate of the hot spot resource pool with the data migration threshold of the hot spot resource pool at an interval of 30S until the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, and migrating the data in the hot spot resource pool to a preset hard disk device for storage.
It can be understood that the invention controls the migration of data in the resource pool by sending a command, and checks whether the data in the hot spot resource pool reaches the roe every 30S, so as to migrate the data, wherein the 30S time can be adjusted at any time, so as to ensure that the performance state of the resource pool is in the optimal state at any time and any place, thereby ensuring the stable operation of the system, and the non-hot spot data can be stored by migrating the data to the non-resource pool, thereby preventing the loss of the data and causing the loss of the data.
In a specific embodiment of the present disclosure, the step S4 is followed by step S5, step S6, step S7 and step S8.
Step S5, calling the record information of the history read-write operation;
step S6, marking abnormal records in the record information of the historical read-write operation to obtain marked record information;
step S7, obtaining a CART decision tree based on a CART algorithm and the marked recording information, carrying out random pruning treatment on the CART decision tree and determining a constant of the CART decision tree, optimizing the decision tree based on the constant of the CART decision tree and a Gini index calculation method to obtain an optimal decision tree, sending the read-write operation of the storage equipment to the optimal decision tree to judge whether the decision tree is abnormal or not, and obtaining abnormal data of the read-write operation of the storage equipment;
and step S8, storing the abnormal data of the read-write operation of the storage device to an abnormal resource pool of a non-hotspot resource pool.
It can be understood that the invention obtains a judgment model of whether the trained data is abnormal or not by marking the historical abnormal information and then training through the CART algorithm, judges the data in the storage device based on the judgment model, and then directly puts the abnormal data into the non-hot resource pool, thereby preventing the data in the hot resource pool from being lost or damaged, and also checking and clearing the abnormal data by sending alarm information to the communication device of the manager.
Example 2
As shown in fig. 2, the present embodiment provides a device for migrating data of a resource pool in a hierarchical manner, where the device includes a first obtaining unit 701, a first sending unit 702, a first calculating unit 703, and a first determining unit 704.
A first obtaining unit 701, configured to obtain first information, second information, and third information, where the first information includes data information input by a user, the second information is structure information of a resource pool, the third information is capacity information of the resource pool, and the data information includes file information, a file size, a read-write mode, and a thread count input by the user;
a first sending unit 702, configured to send the second information to a layering module, so as to obtain a hot spot resource pool and at least one non-hot spot resource pool, where the layering module is a module that performs layering on a resource pool structure;
the first calculating unit 703 is configured to send the first information to a calculating module to obtain a data migration threshold of the hotspot resource pool, where the calculating module is a module for calculating the data migration threshold of the hotspot resource pool;
a first determining unit 704, configured to determine, based on the first information and the third information, whether the usage rate of the hotspot resource pool is greater than a data migration threshold of the hotspot resource pool, to obtain a determination result, and send a command of data migration of the hotspot resource pool based on the determination result.
In a specific embodiment of the present disclosure, the first sending unit 702 includes a first invoking subunit 7021, a first processing subunit 7022, and a second processing subunit 7023.
A first calling subunit 7021, configured to call preset hard disk device information;
a first processing subunit 7022, configured to map structure information of the resource pool based on the preset hard disk device information, where solid state disk information is mapped to a first sub-resource pool in the resource pool, and traditional hard disk information is mapped to a second sub-resource pool in the resource pool, where the first sub-resource pool is a resource pool that receives hot spot data, and the second sub-resource pool is a resource pool that receives migration data of the first sub-resource pool;
a second processing subunit 7023, configured to modify the structures of the first sub-resource pool and the second sub-resource pool based on a preset resource pool type, where an attribute type of the first sub-resource pool is determined, an attribute type of at least one second sub-resource pool is determined, and the corresponding structures of the first sub-resource pool and the at least one second sub-resource pool are modified based on the determined attribute types, so as to obtain a hot resource pool and at least one non-hot resource pool.
In a specific embodiment of the present disclosure, the first calculating unit 703 includes a second invoking subunit 7031, a third processing subunit 7032, and a fourth processing subunit 7033.
A second calling subunit 7031, configured to call data information input by the user in the first information;
a third processing subunit 7032, configured to create multiple execution threads based on the data information input by the user, where the execution threads are created according to the number of threads in the data information input by the user, perform read-write operation on the read-write manner, record file size information of each read-write operation and completion time of each read-write operation in the read-write process, and determine a read-write performance value of the storage device under the condition of the usage rate of the hotspot resource pool according to a preset read-write performance table;
a fourth processing subunit 7033, configured to perform clustering processing on the read-write performance values of the storage device under the condition of the usage rate of the hot spot resource pool based on a distance-based clustering algorithm to obtain at least one read-write performance value cluster, where a distance from a point of each cluster to a cluster center point is obtained by calculation based on an euclidean distance calculation formula, an average value of the distances is calculated, and the average value is used as a data migration threshold of the hot spot resource pool.
In a specific embodiment of the present disclosure, the first determining unit 704 includes a third invoking sub-unit 7041, a fifth processing sub-unit 7042, and a sixth processing sub-unit 7043.
A third invoking subunit 7041, configured to invoke a data migration threshold of the hotspot resource pool;
a fifth processing subunit 7042, configured to calculate a ratio of the first information to the third information based on a capacity utilization formula of a resource pool, to obtain a utilization of the hotspot resource pool;
a sixth processing subunit 7043, configured to compare the usage rate of the hot spot resource pool with the data migration threshold of the resource pool, determine whether the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, and obtain determination result information.
In a specific embodiment of the present disclosure, the first determining unit 704 further includes a first determining subunit 7044 and a second determining subunit 7045.
A first determining subunit 7044, configured to send a first subcommand if the usage rate of the hotspot resource pool is greater than or equal to the data migration threshold of the hotspot resource pool, where the first subcommand is to migrate data in the hotspot resource pool to a preset hard disk device for storage;
a second determining subunit 7045, configured to send a second subcommand if the usage rate of the hot spot resource pool is less than the data migration threshold of the resource pool, where the second subcommand compares the usage rate of the hot spot resource pool with the data migration threshold of the hot spot resource pool at an interval of 30S until the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, and migrates the data in the hot spot resource pool into a preset hard disk device for storage.
In a specific embodiment of the present disclosure, the first determining unit 704 includes a first calling unit 705, a first processing unit 706, a second processing unit 707, and a third processing unit 708.
A first calling unit 705, configured to call record information of a history read-write operation;
a first processing unit 706, configured to mark an abnormal record in the record information of the historical read/write operation, to obtain marked record information;
a second processing unit 707, configured to obtain a CART decision tree based on a CART algorithm and the marked record information, perform random pruning on the CART decision tree, determine a constant of the CART decision tree, optimize the decision tree based on the constant of the CART decision tree and a kini index calculation method, obtain an optimal decision tree, send a read-write operation of a storage device to the optimal decision tree, determine whether the read-write operation is abnormal, and obtain abnormal data of the read-write operation of the storage device;
the third processing unit 708 is configured to store the abnormal data of the read-write operation of the storage device to an abnormal resource pool of the non-hotspot resource pool.
It should be noted that, regarding the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.
Example 3
Corresponding to the above method embodiment, the embodiment of the present disclosure further provides a data layered migration device of a resource pool, where the data layered migration device of a resource pool described below and the data layered migration method of a resource pool described above may be referred to correspondingly.
FIG. 3 is a block diagram illustrating a data hierarchical migration apparatus 800 of a resource pool in accordance with an exemplary embodiment. As shown in fig. 3, the data hierarchical migration apparatus 800 of the resource pool may include: a processor 801, a memory 802. The data layered migration apparatus 800 of the resource pool may further include one or more of a multimedia component 803, an input/output (I/O) interface 804, and a communication component 805.
The processor 801 is configured to control the overall operation of the data hierarchical migration apparatus 800 of the resource pool, so as to complete all or part of the steps in the data hierarchical migration method of the resource pool. The memory 802 is used to store various types of data to support the operation of the data hierarchical migration apparatus 800 in the resource pool, which may include, for example, instructions for any application or method operating on the data hierarchical migration apparatus 800 in the resource pool, as well as application-related data, such as contact data, transceived messages, pictures, audio, video, and so forth. The Memory 802 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. The multimedia components 803 may include screen and audio components. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may further be stored in the memory 802 or transmitted through the communication component 805. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 804 provides an interface between the processor 801 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the data hierarchical migration device 800 and other devices of the resource pool. Wireless communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them, so that the corresponding communication component 805 may include: Wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the data layer migration Device 800 of the resource pool may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, and is used for executing one of the above-mentioned data layer migration methods of the resource pool.
In another exemplary embodiment, a computer readable storage medium is also provided, which comprises program instructions, which when executed by a processor, implement the steps of the above-described method for data hierarchical migration of a resource pool. For example, the computer readable storage medium may be the memory 802 described above comprising program instructions executable by the processor 801 of the data hierarchical migration apparatus 800 of the resource pool to perform the data hierarchical migration method of the resource pool described above.
Example 4
Corresponding to the above method embodiment, the present disclosure further provides a readable storage medium, and a readable storage medium described below and a data hierarchical migration method of a resource pool described above may be referred to correspondingly.
A readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the data hierarchical migration method of a resource pool of the above-mentioned method embodiments.
The readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various readable storage media capable of storing program codes.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for layered migration of data in a resource pool is characterized by comprising the following steps:
acquiring first information, second information and third information, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, a read-write mode and thread number input by the user;
sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool, wherein the layering module is a module for layering the resource pool structure;
sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, wherein the computing module is a module for computing the data migration threshold of the hotspot resource pool;
and judging whether the utilization rate of the hot spot resource pool is greater than the data migration threshold value of the hot spot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a data migration command of the hot spot resource pool based on the judgment result.
2. The method for the layered migration of data in a resource pool according to claim 1, wherein the step of sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool comprises:
calling preset hard disk equipment information;
mapping the structural information of the resource pool based on the preset hard disk device information, wherein the solid state disk information is mapped to a first sub-resource pool in the resource pool, and the traditional hard disk information is mapped to a second sub-resource pool in the resource pool, wherein the first sub-resource pool is a resource pool for receiving hot spot data, and the second sub-resource pool is a resource pool for receiving migration data of the first sub-resource pool;
and modifying the structures of the first sub resource pool and the second sub resource pool based on a preset resource pool type, wherein the attribute type of the first sub resource pool is determined, the attribute type of at least one second sub resource pool is determined, and the corresponding structures of the first sub resource pool and the at least one second sub resource pool are modified based on the determined attribute types to obtain a hot resource pool and at least one non-hot resource pool.
3. The method for the layered migration of data in a resource pool according to claim 1, wherein the step of sending the first information to a computing module to obtain the data migration threshold of the hotspot resource pool comprises:
calling data information input by a user in the first information;
creating a plurality of execution threads based on the data information input by the user, wherein the execution threads are correspondingly created according to the thread number in the data information input by the user, performing read-write operation on the read-write mode, recording the file size information of each read-write operation and the completion time of each read-write operation in the read-write process, and determining the read-write performance value of the storage device under the condition of the utilization rate of the hot resource pool according to a preset read-write performance table;
clustering the read-write performance values of the storage device under the condition of the utilization rate of the hot spot resource pool based on a distance clustering algorithm to obtain at least one read-write performance value cluster, wherein the distance from a point of each cluster to a cluster central point is calculated based on a Euclidean distance calculation formula, the average value of the distances is calculated, and the average value is used as a data migration threshold value of the hot spot resource pool.
4. The method for the layered data migration of the resource pool according to claim 1, wherein determining whether the usage rate of the hot spot resource pool is greater than the data migration threshold of the hot spot resource pool based on the first information and the third information includes:
calling a data migration threshold of the hot spot resource pool;
calculating the ratio of the first information to the third information based on a capacity utilization rate formula of a resource pool to obtain the utilization rate of the hotspot resource pool;
and comparing the utilization rate of the hot spot resource pool with the data migration threshold of the resource pool, and judging whether the utilization rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool to obtain judgment result information.
5. An apparatus for layered migration of data from a resource pool, comprising:
the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first information comprises data information input by a user, the second information is structural information of a resource pool, the third information is capacity information of the resource pool, and the data information comprises file information, file size, reading and writing modes and thread number input by the user;
the first sending unit is used for sending the second information to a layering module to obtain a hot spot resource pool and at least one non-hot spot resource pool, and the layering module is a module for layering the resource pool structure;
the first computing unit is used for sending the first information to a computing module to obtain a data migration threshold of the hotspot resource pool, and the computing module is a module for computing the data migration threshold of the hotspot resource pool;
and the first judging unit is used for judging whether the utilization rate of the hot resource pool is greater than the data migration threshold of the hot resource pool or not based on the first information and the third information to obtain a judgment result, and sending a hot resource pool data migration command based on the judgment result.
6. The apparatus for hierarchical migration of data from a resource pool according to claim 5, wherein the apparatus comprises:
the first calling subunit is used for calling preset hard disk equipment information;
the first processing subunit is configured to map structural information of the resource pool based on the preset hard disk device information, wherein the solid state disk information is mapped to a first sub-resource pool in the resource pool, and the conventional hard disk information is mapped to a second sub-resource pool in the resource pool, the first sub-resource pool is a resource pool for receiving hot spot data, and the second sub-resource pool is a resource pool for receiving migration data of the first sub-resource pool;
and the second processing subunit is configured to modify the structures of the first sub-resource pool and the second sub-resource pool based on a preset resource pool type, determine an attribute type of the first sub-resource pool and determine an attribute type of at least one second sub-resource pool, and modify the structures of the corresponding first sub-resource pool and the corresponding second sub-resource pool based on the determined attribute types to obtain a hot spot resource pool and at least one non-hot spot resource pool.
7. The apparatus for data hierarchical migration of resource pools according to claim 5, wherein the apparatus comprises:
the second calling subunit is used for calling the data information input by the user in the first information;
a third processing subunit, configured to create multiple execution threads based on the data information input by the user, where an execution thread is created according to a thread number in the data information input by the user, perform read-write operation on the read-write mode, record file size information of each read-write operation and completion time of each read-write operation in a read-write process, and determine a read-write performance value of the storage device under the condition of the usage rate of the hotspot resource pool according to a preset read-write performance table;
and the fourth processing subunit is used for clustering the read-write performance values of the storage device under the condition of the utilization rate of the hot spot resource pool based on a distance clustering algorithm to obtain at least one read-write performance value cluster, wherein the distance from each cluster point to a cluster central point is calculated based on an Euclidean distance calculation formula, the average value of the distances is calculated, and the average value is used as the data migration threshold value of the hot spot resource pool.
8. The apparatus for layered data migration of a resource pool according to claim 5, wherein determining whether the usage rate of the hot spot resource pool is greater than the data migration threshold of the hot spot resource pool based on the first information and the third information comprises:
the third calling subunit is used for calling the data migration threshold of the hotspot resource pool;
the fifth processing subunit is configured to calculate a ratio of the first information to the third information based on a capacity utilization formula of a resource pool, so as to obtain a utilization rate of the hotspot resource pool;
and the sixth processing subunit is configured to compare the usage rate of the hot spot resource pool with the data migration threshold of the resource pool, determine whether the usage rate of the hot spot resource pool is greater than or equal to the data migration threshold of the hot spot resource pool, and obtain determination result information.
9. A device for hierarchical migration of data from a resource pool, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data hierarchical migration method of the resource pool according to any one of claims 1 to 4 when executing the computer program.
10. A readable storage medium, characterized by: the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the method for data hierarchical migration of a resource pool according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210483991.8A CN114936003B (en) | 2022-05-06 | 2022-05-06 | Data layered migration method, device and equipment of resource pool and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210483991.8A CN114936003B (en) | 2022-05-06 | 2022-05-06 | Data layered migration method, device and equipment of resource pool and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114936003A true CN114936003A (en) | 2022-08-23 |
| CN114936003B CN114936003B (en) | 2023-03-21 |
Family
ID=82865243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210483991.8A Active CN114936003B (en) | 2022-05-06 | 2022-05-06 | Data layered migration method, device and equipment of resource pool and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114936003B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160350292A1 (en) * | 2015-05-27 | 2016-12-01 | Alibaba Group Holding Limited | Method and apparatus for real-time data migration |
| CN106502578A (en) * | 2015-09-06 | 2017-03-15 | 中兴通讯股份有限公司 | Capacity change suggesting method and device |
| CN106502576A (en) * | 2015-09-06 | 2017-03-15 | 中兴通讯股份有限公司 | Migration strategy method of adjustment, capacity change suggesting method and device |
| CN110688067A (en) * | 2019-09-06 | 2020-01-14 | 苏州浪潮智能科技有限公司 | Layering method, device, equipment and medium of storage system |
| CN113836116A (en) * | 2021-09-29 | 2021-12-24 | 济南浪潮数据技术有限公司 | Data migration method and device, electronic equipment and readable storage medium |
-
2022
- 2022-05-06 CN CN202210483991.8A patent/CN114936003B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160350292A1 (en) * | 2015-05-27 | 2016-12-01 | Alibaba Group Holding Limited | Method and apparatus for real-time data migration |
| CN106502578A (en) * | 2015-09-06 | 2017-03-15 | 中兴通讯股份有限公司 | Capacity change suggesting method and device |
| CN106502576A (en) * | 2015-09-06 | 2017-03-15 | 中兴通讯股份有限公司 | Migration strategy method of adjustment, capacity change suggesting method and device |
| CN110688067A (en) * | 2019-09-06 | 2020-01-14 | 苏州浪潮智能科技有限公司 | Layering method, device, equipment and medium of storage system |
| CN113836116A (en) * | 2021-09-29 | 2021-12-24 | 济南浪潮数据技术有限公司 | Data migration method and device, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114936003B (en) | 2023-03-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105740078B (en) | Memory management method, device and terminal | |
| CN109614045B (en) | Metadata dropping method and device and related equipment | |
| CN107734183A (en) | A kind of method, storage medium and the mobile terminal of one-handed performance mobile terminal | |
| CN108874324B (en) | Access request processing method, device, equipment and readable storage medium | |
| CN106802772A (en) | The method of data record, device and solid state hard disc | |
| CN110795400A (en) | File management method, device, equipment and medium | |
| CN114328281A (en) | Method and device for processing abnormal power failure of solid state disk, electronic equipment and medium | |
| CN114968839A (en) | Hard disk garbage recycling method, device and equipment and computer readable storage medium | |
| US9785374B2 (en) | Storage device management in computing systems | |
| CN115145391A (en) | Virtual reality treadmill self-adaption method and system based on AI deep learning | |
| CN114996173A (en) | Method and device for managing write operation of storage equipment | |
| CN118331513A (en) | Data intelligent dynamic scheduling method and device and computer equipment | |
| CN114936003B (en) | Data layered migration method, device and equipment of resource pool and readable storage medium | |
| CN112000971B (en) | File permission recording method, system and related device | |
| CN107992361A (en) | Method for cleaning, device, storage medium and the electronic equipment of background application | |
| CN117311628A (en) | Command processing method, device, equipment and medium of solid state disk | |
| CN108205559A (en) | A kind of data managing method and its equipment | |
| CN115514775A (en) | Data processing method, device, equipment and storage medium | |
| CN115185458A (en) | Data writing method and device, storage medium and computer equipment | |
| CN114942825A (en) | Virtual machine live migration method and device, electronic equipment and storage medium | |
| CN107623784A (en) | Method for information display and device, terminal installation and computer-readable recording medium | |
| CN117666958B (en) | Method, device, equipment and medium for prolonging storage life of identification module | |
| CN114489519B (en) | IO request processing method, device, equipment and readable storage medium | |
| CN114879916B (en) | Method and device for managing storage unit | |
| CN114968595B (en) | Processing method, device, computer equipment and storage medium of NFT owner information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |