CN114442952A - Cold data migration method and device, storage medium and electronic device - Google Patents
Cold data migration method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN114442952A CN114442952A CN202210090172.7A CN202210090172A CN114442952A CN 114442952 A CN114442952 A CN 114442952A CN 202210090172 A CN202210090172 A CN 202210090172A CN 114442952 A CN114442952 A CN 114442952A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- cold
- attribute
- hard disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 84
- 238000013508 migration Methods 0.000 title claims abstract description 84
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000004590 computer program Methods 0.000 claims description 16
- 238000011084 recovery Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 7
- 230000003068 static effect Effects 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000004048 modification Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000032683 aging Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010438 heat treatment Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a cold data migration method, a cold data migration device, a storage medium and an electronic device, wherein the method comprises the following steps: acquiring attribute information of target data stored in a first hard disk in a target server under the condition that the first space occupancy rate of the first hard disk is greater than a first threshold value; determining target cold data to be migrated included in the target data based on the attribute information; and migrating the target cold data to a second hard disk in the target server, wherein the second space occupancy rate of the second hard disk is less than a second threshold value. By the method and the device, the problem that data query and large-scale storage data distribution are unreasonable in the related technology is solved, and the effect of reasonably distributing the data query and the large-scale storage data is achieved.
Description
Technical Field
The embodiment of the invention relates to the field of communication, in particular to a cold data migration method, a cold data migration device, a storage medium and an electronic device.
Background
In a big data scenario, the elastic search is usually used as an engine or a database for big data query and search because of its features of near real-time query and distributed storage of a large amount of data. The Elasticsearch can have this capability, relying not only on its excellent system design, but also on the underlying storage medium. Usually, the data of the Elasticsearch is stored in an SSD (Solid State Disk), but the SSD has a small Disk capacity and a high cost, and a user wants to store as much data as possible under a certain budget, which is inconsistent with the above. In addition, most data have certain timeliness, the query and writing of a large data volume scene often aim at data (hot data) in a recent period, and the query frequency of the data in the rest period is very small or almost not checked, and the writing pressure is small (cold data). The part of data users want to reserve and inquire a little, but under the condition of certain storage resources, the real-time data storage space is smaller and smaller.
Therefore, the problem that data query and large-scale storage data distribution are unreasonable exists in the related art.
In view of the above problems in the related art, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a cold data migration method, a cold data migration device, a storage medium and an electronic device, and aims to at least solve the problem that data query and large-scale storage data distribution are unreasonable in the related technology.
According to an embodiment of the present invention, there is provided a cold data migration method including: acquiring attribute information of target data stored in a first hard disk in a target server under the condition that a first space occupancy rate of the first hard disk is greater than a first threshold; determining target cold data to be migrated included in the target data based on the attribute information; and migrating the target cold data to a second hard disk in the target server, wherein the second space occupancy rate of the second hard disk is smaller than a second threshold value.
According to another embodiment of the present invention, there is provided a migration apparatus of cold data, including: the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring attribute information of target data stored in a first hard disk in a target server under the condition that a first space occupancy rate of the first hard disk is greater than a first threshold; the determining module is used for determining target cold data to be migrated in the target data based on the attribute information; and the migration module is used for migrating the target cold data to a second hard disk in the target server, wherein the second space occupancy rate of the second hard disk is smaller than a second threshold value.
According to yet another embodiment of the invention, there is also provided a computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, performs the steps of the method as set forth in any one of the above.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
According to the method and the device, under the condition that the first space occupancy rate of the first hard disk in the target server is larger than the first threshold value, the attribute information of the target data existing in the first hard disk is obtained, the target cold data to be migrated in the target data is determined according to the attribute information, and the target cold data is migrated to the second hard disk in the target server. Under the condition that the first space occupancy rate of the first hard disk is greater than the first threshold, the target cold data can be migrated to the second hard disk of which the second space occupancy rate is less than the second threshold in the target server, and the space occupancy rate of the first hard disk is released, so that the first hard disk can store more other data and can meet the rapid query of the data, therefore, the problem that the data query and the large-scale storage data distribution in the related technology are unreasonable can be solved, and the effects of reasonably distributing the data query and the large-scale storage data are achieved.
Drawings
Fig. 1 is a block diagram of a hardware configuration of a mobile terminal of a cold data migration method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of migration of cold data according to an embodiment of the present invention;
FIG. 3 is a diagram of Es node and service relationships, according to an illustrative embodiment of the invention;
FIG. 4 is a flow diagram of a method for migration of cold data, in accordance with a specific embodiment of the present invention;
FIG. 5 is a flowchart of Es index migration, according to a specific embodiment of the present invention;
fig. 6 is a block diagram of a migration apparatus of cold data according to an embodiment of the present invention.
Detailed Description
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings in conjunction with the embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking an example of the method running on a mobile terminal, fig. 1 is a hardware structure block diagram of the mobile terminal of a cold data migration method according to an embodiment of the present invention. As shown in fig. 1, the mobile terminal may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, wherein the mobile terminal may further include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store a computer program, for example, a software program of an application software and a module, such as a computer program corresponding to the cold data migration method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In this embodiment, a cold data migration method is provided, and fig. 2 is a flowchart of a cold data migration method according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, acquiring attribute information of target data stored in a first hard disk in a target server under the condition that a first space occupancy rate of the first hard disk is greater than a first threshold;
step S204, determining target cold data to be migrated in the target data based on the attribute information;
step S206, migrating the target cold data to a second hard disk in the target server, wherein a second space occupancy rate of the second hard disk is smaller than a second threshold.
In the above embodiment, the first hard disk may be an SSD solid state disk, the second hard disk may be a SATA hard disk, and the target server may adopt a mode of mixing the SSD and the SATA. The SSD has high cost and high reading and writing speed, and can store hot data; the SATA is low in cost and low in read-write speed, cold data can be stored, and data query and large-scale storage reasonable distribution can be achieved at a certain cost by hybrid mounting of the SSD and the SATA on the same server. In the target server, the capacity of the SSD may be 1.5T, the capacity of the SATA may be 2T, the memory may be 125G, and the CPU may be 32 cores. It should be noted that the capacities of the SSD, the SATA and the memory may be other capacities, and the CPU may be 64 cores, etc., which is not limited in this invention.
In the above embodiment, the second hard disk may be one or multiple, for example, multiple SATA hard disks may be configured in the target server. When the number of the second hard disks is multiple, the target second hard disk for storing the target cold data to be migrated may be selected from the second hard disks, and when the target second hard disk is determined, the hard disk with the minimum space occupancy rate in the second hard disks may be determined as the target second hard disk, or any one of the hard disks with the space occupancy rate smaller than the second threshold included in the second hard disk may be determined as the target second hard disk.
In the above-described embodiment, the attribute information may be information created when the data is stored, and the index of the target data may be created according to the type of the data. For example, indexes are created according to whether data has timeliness, only one index without timeliness is created, an index with timeliness is created according to time in a fixed time period, and attribute information of the index can be suffixed with time (e.g., xxxx-2021.07). Attributes in the index may add index. SSD, the newly created index is placed on the SSD, and the introduced aging data is initially on the SSD, so that the performance of data query is ensured.
In the above embodiment, the first threshold may be 80%, and the second threshold may be 85%, when the space of the second hard disk is insufficient, the migration is not performed, and when the space occupancy rate of the first hard disk exceeds the first threshold, the data migration is performed. During the data migration, the first space occupancy rate of the first hard disk can be detected in real time, and the migration can be stopped when the first space occupancy rate is smaller than the fourth threshold value. Wherein the fourth threshold may be 75%. It should be noted that the first threshold, the second threshold, and the fourth threshold are only an exemplary illustration, and all of the thresholds support profile modification, that is, a user may modify the thresholds in a customized manner, and the present invention does not limit the thresholds.
Optionally, the main body of the above steps may be a background processor or other devices with similar processing capabilities, and may also be a machine integrated with at least a data processing device, where the data processing device may include a terminal such as a computer, a mobile phone, and the like, but is not limited thereto.
According to the method and the device, under the condition that the first space occupancy rate of the first hard disk in the target server is larger than the first threshold value, the attribute information of the target data existing in the first hard disk is obtained, the target cold data to be migrated in the target data is determined according to the attribute information, and the target cold data is migrated to the second hard disk in the target server. Under the condition that the first space occupancy rate of the first hard disk is greater than the first threshold, the target cold data can be migrated to the second hard disk of which the second space occupancy rate is less than the second threshold in the target server, and the space occupancy rate of the first hard disk is released, so that the first hard disk can store more other data and can meet the rapid query of the data, therefore, the problem that the data query and the large-scale storage data distribution in the related technology are unreasonable can be solved, and the effects of reasonably distributing the data query and the large-scale storage data are achieved.
In an exemplary embodiment, migrating the target cold data to a second hard disk in the target server comprises: determining the target number of nodes in a cluster under the condition that the cluster where the target cold data is located is in a normal working state and the cluster comprises cold nodes; determining a target migration rule for the target cold data based on the target quantity; migrating the target cold data based on the target migration rules. In this embodiment, the characteristics of optimal 31G of the elastic search heap memory can be utilized to deploy 2 Es nodes, one hot data node and one cold data node on the target server; the data directories are respectively placed on the SSD and the SATA; cold and hot nodes are distinguished by node name (name) and node attribute (node. attr. type); the cold nodes and the hot nodes belong to the same cluster. When the cluster where the target cold data is located is in a healthy state, namely a normal working state, and the cluster includes the cold nodes, the target number of the nodes included in the cluster can be determined, the target migration rule is determined according to the target number, and the target cold data is migrated based on the target migration rule. The Es node and service relationship diagram can be seen in fig. 3.
In one exemplary embodiment, determining the target migration rule for the target cold data based on the target quantity comprises: determining the target migration rules corresponding to the target quantity from the corresponding relation between the quantity configured in advance and the migration rules; wherein the corresponding relationship comprises: in the case that the number is greater than the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to be a first attribute, and setting the attribute of the target index of the cold data to be a second attribute; in the case that the number is less than or equal to the third threshold, the corresponding migration rule includes: and updating the attribute of the cold node to be a third attribute, and setting the attribute of the target index of the cold data to be the second attribute. In this embodiment, data is first stored in the first hard disk, and when the size of the data in the first hard disk is accumulated to a certain amount, the cluster SSD gradually becomes full of space, and the data is little or no queried earlier, and the data becomes cold data. And the cold-hot migration management program detects the size of the disk at regular time, and starts to execute cold-hot migration after reaching a certain threshold, cold data are gradually migrated to a cold node, the SSD disk space is released, and the normal import and query of new aging data are ensured. The target migration rule for the target cold data may be determined according to a target number of nodes included in the cluster in which the cold node is located. That is, the target migration rule corresponding to the target number may be determined according to the correspondence between the node number and the migration rule. When the number of nodes is greater than the third threshold, the migration rule may be to update the attribute of the cold node to the first attribute and set the attribute of the target index of the cold data to the second attribute. The first attribute may be track, and the second attribute may be sata. Namely, the attribute of cluster, routing, allocation, artifact, attribute is track, and the attribute of index, routing, allocation, attribute is sata. And when the number of the nodes is less than or equal to a third threshold value, updating the attribute of the cold node to be a third attribute, and setting the attribute of the target index of the cold data to be a second attribute. Namely, the attribute of cluster. The third threshold may be 4, that is, when determining the target migration rule, it may be first determined whether the number of nodes in the cluster exceeds 4, and when the number of nodes exceeds 4, update the attribute of cluster. It should be noted that the third threshold is only an exemplary illustration, and the third threshold may also be set to 2,3,5,6, etc., which is not limited in the present invention. When the third threshold value is 4, adopting different migration methods according to the number of the nodes of the cluster, wherein the execution methods of the clusters below 4 nodes are different from those of the clusters above 4 nodes; the migration rules may be as shown in table 1:
TABLE 1
In one exemplary embodiment, migrating the target cold data based on the target migration rules comprises: reducing the adjustment of the index concurrency recovery number of the cluster; migrating a target index of the target cold data based on the target migration rule with the reduced number of index concurrency recoveries. In this embodiment, when migrating the target cold data, the number of index concurrency recovery of the cluster may be reduced, and after the number of index concurrency recovery of the cluster is reduced, the target cold data is migrated. For example, the index concurrency recovery number (node _ current _ recovery) is reduced to a predetermined number. Reducing the number of index concurrency recovery of the cluster can reduce the disk io of the SATA. The predetermined number may be 2 (the number is only an exemplary illustration, and the predetermined number may also be 1, 3, etc., which is not limited by the present invention). After the data migration is complete, the number of concurrent index recoveries may be restored to a default value, such as 24.
In one exemplary embodiment, determining target cold data to be migrated included in the target data based on the attribute information includes: determining that the attribute information included in the target data is first data of a fourth attribute; determining the target cold data from the first data based on time information included in the fourth attribute, wherein the time information represents a time when the first data is stored. In this embodiment, when determining the target cold data, it may be determined that the attribute information included in the target data is the first number of the fourth attribute, where the fourth attribute may be an attribute including ssd. And determining the target cold data from the first data according to the time information included in the fourth attribute. Wherein the time information may be time information for creating the data index.
In one exemplary embodiment, determining the target cold data from the first data based on the time information included in the fourth attribute includes: determining a creation time of the first data based on time information included in the fourth attribute; and determining data, included in the first data, having a time difference between a creation time and a current time greater than a predetermined threshold as the target cold data. In this embodiment, when determining the target cold data, the creation time of the first data may be determined according to the time information included in the fourth attribute, and data having a time difference between the creation time and the current time greater than a predetermined threshold may be determined as the target cold data. Wherein the predetermined threshold may be 1 week (or other threshold, such as 5 days, 10 days, etc.). That is, the previously stored time may be migrated to the second hard disk first in chronological order.
In one exemplary embodiment, the method further comprises: determining the data type of the acquired data; configuring the attribute information of the data to include time information and attribute information of a hot data type in a case where the data type indicates that the data is time-sensitive data; configuring the attribute information of the data to include attribute information of a static data type and a hot data type, in case the data type indicates that the data is non-time sensitive data. In this embodiment, after the data may be acquired, the data type of the acquired data may be determined, and if the data is time-efficient data, the attribute information of the data may be configured to include the attribute information of the time information and the thermal data type, and if the data is non-time-efficient data, the attribute information of the data may be configured to include the attribute information of the static data type and the thermal data type. Where the thermal data type may be represented as ssd. When SSD is included in the attribute information, it indicates that the data is stored in the SSD.
The method for migrating cold data is described below with reference to specific embodiments:
fig. 4 is a flowchart of a cold data migration method according to an embodiment of the present invention, where the flowchart includes:
step S402, cluster disk information is acquired. When the process starts, the program checks the connection state of the Es client, if the connection fails, the connection is detected once every 10min (configurable), the connection is not successful after 3h (configurable), and the migration operation is abandoned; if the connection is successful, the process continues to the next step.
Step S404 determines whether or not a cold data node is included, and if the determination result is yes, step S406 is executed, and if the determination result is no, step S412 is executed. Whether the cluster contains the cold data nodes is detected before each migration operation, and cold and hot migration failure caused by environment change or misoperation of the field cluster is prevented. When the historical version without the cold disk is upgraded to the version with the cold disk, after the console is started, the historical index is added with the index. ssd, index shards are distributed across hot data nodes.
In step S406, all indexes are acquired.
Step S408, detecting a cold disc space.
Step S410 is to determine whether or not the cold disk space (corresponding to the second hard disk) exceeds a threshold value, and if yes, step S412 is executed, and if no, step S414 is executed. The cold tray space threshold (corresponding to the second threshold) is 85%, and when the cold tray space is insufficient, no migration is performed;
step S412, the task is ended.
In step S414, a hot plate space is detected.
In step S416, it is determined whether or not the hot disk space (corresponding to the first hard disk) exceeds the threshold, and if the determination result is yes, step S418 is executed, and if the determination result is no, step S412 is executed. The hot-disk space threshold (corresponding to the first threshold) is 80%, and when the hot-disk space exceeds the threshold, the data migration is performed, wherein the hot-disk space threshold supports the modification of the configuration file.
In step S418, Es is transferred by cooling and heating.
Step S420 determines whether or not the execution is completed, and if the determination result is yes, step S422 is executed, and if the determination result is no, step S418 is executed.
In step S422, a target threshold value (corresponding to the fourth threshold value) is set. The target threshold is 75% by default, configuration file modification is supported, and the hot disk space can continuously migrate below the target threshold.
Fig. 5 is a flowchart illustrating Es index migration according to an embodiment of the present invention, and as shown in fig. 5, the program may select an index with the earliest time in the time series index each time, and may determine the index to be migrated according to the time suffix judgment (xxxx-2021.01) in the index name, or according to the prefix comprehensive judgment when a cluster includes indexes with the same suffixes. Judging whether the cluster state is healthy, and reducing the index concurrency recovery number (node _ current _ recovery) to 2 to reduce the disk io of the SATA in case of the healthy condition. Different migration methods are adopted according to the number of the nodes of the cluster, and the execution methods of the clusters below 4 nodes and above 4 nodes are different, and the specific mode can be seen in table 1. After the method is executed, the index is in a migration state, and the program needs to monitor the migration situation at regular time. And restoring the concurrent recovery number after the index migration is completed.
In the foregoing embodiment, the SSD and the SATA disk are mounted on the same server in a mixed manner, and when a cluster is deployed, cold and hot nodes are deployed on different storage media of the same server, so that cold and hot data are separated from hot data on the storage media; the requirement of storing data as much as possible under a certain budget cost of a client is met. By means of establishing the aging data in the same time range in the same index, management of cold data and hot data is achieved, and therefore most of cold data can be separated from hot data.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
In this embodiment, a cold data migration apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted for brevity. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 6 is a block diagram of a cold data migration apparatus according to an embodiment of the present invention, as shown in fig. 6, the apparatus including:
the acquiring module 62 is configured to acquire attribute information of target data stored in a first hard disk in a target server when a first space occupancy rate of the first hard disk is greater than a first threshold;
a determining module 64, configured to determine, based on the attribute information, target cold data to be migrated included in the target data;
a migration module 66, configured to migrate the target cold data to a second hard disk in the target server, where a second space occupancy rate of the second hard disk is smaller than a second threshold.
In an exemplary embodiment, the migration module 66 may implement the migration of the target cold data to the second hard disk in the target server by: determining the target number of nodes in a cluster under the condition that the cluster where the target cold data is located is in a normal working state and the cluster comprises cold nodes; determining a target migration rule for the target cold data based on the target quantity; migrating the target cold data based on the target migration rules.
In an exemplary embodiment, migration module 66 may implement the target migration rule to determine the target cold data based on the target quantity by: determining the target migration rules corresponding to the target quantity from the corresponding relation between the quantity configured in advance and the migration rules; wherein the corresponding relationship comprises: in the case that the number is greater than the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to be a first attribute, and setting the attribute of the target index of the cold data to be a second attribute; in the case that the number is less than or equal to the third threshold, the corresponding migration rule includes: and updating the attribute of the cold node to be a third attribute, and setting the attribute of the target index of the cold data to be the second attribute.
In an exemplary embodiment, the migration module 66 may effect the migration of the target cold data based on the target migration rule by: reducing the index concurrency recovery number of the cluster; migrating a target index of the target cold data based on the target migration rule with the reduced number of index concurrency recoveries.
In an exemplary embodiment, the determining module 64 may determine the target cold data to be migrated included in the target data based on the attribute information by: determining that the attribute information included in the target data is first data of a fourth attribute; determining the target cold data from the first data based on time information included in the fourth attribute, wherein the time information represents a time when the first data is stored.
In an exemplary embodiment, the determining module 64 may determine the target cold data from the first data based on the time information included in the fourth attribute by: determining a creation time of the first data based on time information included in the fourth attribute; and determining data, included in the first data, having a time difference between a creation time and a current time greater than a predetermined threshold as the target cold data.
In an example embodiment, the apparatus may be configured to determine a data type of the acquired data; configuring the attribute information of the data to include time information and attribute information of a hot data type in a case where the data type indicates that the data is time-sensitive data; configuring the attribute information of the data to include attribute information of a static data type and a hot data type, in case the data type indicates that the data is non-time sensitive data.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method as set forth in any of the above.
In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
In an exemplary embodiment, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary embodiments, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the various modules or steps of the invention described above may be implemented using a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and they may be implemented using program code executable by the computing devices, such that they may be stored in a memory device and executed by the computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into various integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for migrating cold data, comprising:
acquiring attribute information of target data stored in a first hard disk in a target server under the condition that a first space occupancy rate of the first hard disk is greater than a first threshold;
determining target cold data to be migrated included in the target data based on the attribute information;
and migrating the target cold data to a second hard disk in the target server, wherein the second space occupancy rate of the second hard disk is smaller than a second threshold value.
2. The method of claim 1, wherein migrating the target cold data to a second hard disk in the target server comprises:
determining the target number of nodes in a cluster under the condition that the cluster where the target cold data is located is in a normal working state and the cluster comprises cold nodes;
determining a target migration rule for the target cold data based on the target quantity;
migrating the target cold data based on the target migration rules.
3. The method of claim 2, wherein determining the target migration rule for the target cold data based on the target quantity comprises:
determining the target migration rules corresponding to the target quantity from the corresponding relation between the quantity configured in advance and the migration rules;
wherein the corresponding relationship comprises:
in the case that the number is greater than the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to be a first attribute, and setting the attribute of the target index of the cold data to be a second attribute;
in the case that the number is less than or equal to the third threshold, the corresponding migration rule includes: and updating the attribute of the cold node to be a third attribute, and setting the attribute of the target index of the cold data to be the second attribute.
4. The method of claim 2, wherein migrating the target cold data based on the target migration rules comprises:
reducing the index concurrency recovery number of the cluster;
migrating a target index of the target cold data based on the target migration rule with the reduced number of index concurrency recoveries.
5. The method of claim 1, wherein determining target cold data to be migrated included in the target data based on the attribute information comprises:
determining that the attribute information included in the target data is first data of a fourth attribute;
determining the target cold data from the first data based on time information included in the fourth attribute, wherein the time information represents a time when the first data is stored.
6. The method of claim 5, wherein determining the target cold data from the first data based on the time information included in the fourth attribute comprises:
determining a creation time of the first data based on time information included in the fourth attribute;
and determining data, included in the first data, of which the time difference between the creation time and the current time is greater than a predetermined threshold value, as the target cold data.
7. The method of claim 1, further comprising:
determining the data type of the acquired data;
configuring the attribute information of the data to include time information and attribute information of a hot data type in a case where the data type indicates that the data is time-sensitive data;
configuring the attribute information of the data to include attribute information of a static data type and a hot data type, in case the data type indicates that the data is non-time sensitive data.
8. A migration apparatus of cold data, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring attribute information of target data stored in a first hard disk in a target server under the condition that a first space occupancy rate of the first hard disk is greater than a first threshold;
a determining module, configured to determine, based on the attribute information, target cold data to be migrated included in the target data;
and the migration module is used for migrating the target cold data to a second hard disk in the target server, wherein the second space occupancy rate of the second hard disk is smaller than a second threshold value.
9. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, wherein the computer program, when being executed by a processor, carries out the steps of the method as claimed in any one of the claims 1 to 7.
10. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210090172.7A CN114442952A (en) | 2022-01-25 | 2022-01-25 | Cold data migration method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210090172.7A CN114442952A (en) | 2022-01-25 | 2022-01-25 | Cold data migration method and device, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114442952A true CN114442952A (en) | 2022-05-06 |
Family
ID=81368840
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210090172.7A Pending CN114442952A (en) | 2022-01-25 | 2022-01-25 | Cold data migration method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114442952A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118170324A (en) * | 2024-05-11 | 2024-06-11 | 中移(苏州)软件技术有限公司 | Cloud hard disk migration method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105843820A (en) * | 2015-01-16 | 2016-08-10 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| CN110196851A (en) * | 2019-05-09 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN111831232A (en) * | 2020-07-22 | 2020-10-27 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| US20210133157A1 (en) * | 2018-11-20 | 2021-05-06 | Shanghaitech University | Data management system, method, terminal and medium based on hybrid storage |
-
2022
- 2022-01-25 CN CN202210090172.7A patent/CN114442952A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105843820A (en) * | 2015-01-16 | 2016-08-10 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| US20210133157A1 (en) * | 2018-11-20 | 2021-05-06 | Shanghaitech University | Data management system, method, terminal and medium based on hybrid storage |
| CN110196851A (en) * | 2019-05-09 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN111831232A (en) * | 2020-07-22 | 2020-10-27 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118170324A (en) * | 2024-05-11 | 2024-06-11 | 中移(苏州)软件技术有限公司 | Cloud hard disk migration method and device, electronic equipment and storage medium |
| CN118170324B (en) * | 2024-05-11 | 2024-08-16 | 中移(苏州)软件技术有限公司 | Cloud hard disk migration method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11573725B2 (en) | Object migration method, device, and system | |
| CN112765262B (en) | Data redistribution method, electronic equipment and storage medium | |
| CN110196851A (en) | A kind of date storage method, device, equipment and storage medium | |
| EP3951607A1 (en) | Data reading method, data writing method, and server | |
| CN104615606A (en) | Hadoop distributed file system and management method thereof | |
| CN108874502B (en) | Resource management method, device and equipment of cloud computing cluster | |
| CN109977099A (en) | Database portion arranging method, user equipment, storage medium and device | |
| CN106302640A (en) | Data request processing method and device | |
| CN111475483A (en) | Database migration method and device and computing equipment | |
| CN110727738A (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| CN113051250A (en) | Database cluster capacity expansion method and device, electronic equipment and storage medium | |
| CN115328924A (en) | Method and device for creating table space, storage medium and electronic equipment | |
| CN114442952A (en) | Cold data migration method and device, storage medium and electronic device | |
| CN114610680A (en) | Method, device and equipment for managing metadata of distributed file system and storage medium | |
| CN101483668A (en) | Network storage and access method, device and system for hot spot data | |
| CN115794305A (en) | Virtual machine memory adjusting method and device, storage medium and electronic device | |
| CN113312328B (en) | Control method, data processing method, data access method and computing device | |
| CN115079958A (en) | Multi-node load balancing cold and hot data migration device, method, terminal and medium | |
| CN115033551A (en) | Database migration method and device, electronic equipment and storage medium | |
| CN104765748A (en) | Method and device for converting copying table into slicing table | |
| CN115827745A (en) | Memory database cluster and implementation method and device thereof | |
| CN112860694B (en) | Service data processing method, device and equipment | |
| CN110874352A (en) | Database management method and system | |
| CN116028234B (en) | Distributed database load balancing method, device, equipment and storage medium | |
| CN117076157B (en) | Request management method, request management device, computer readable storage medium and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |