CN113835636A - Data storage method and device, full-flash equipment and computer readable storage medium - Google Patents
Data storage method and device, full-flash equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113835636A CN113835636A CN202111124219.9A CN202111124219A CN113835636A CN 113835636 A CN113835636 A CN 113835636A CN 202111124219 A CN202111124219 A CN 202111124219A CN 113835636 A CN113835636 A CN 113835636A
- Authority
- CN
- China
- Prior art keywords
- metadata
- volatile memory
- memory
- persistent memory
- preset threshold
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000013500 data storage Methods 0.000 title claims abstract description 35
- 230000015654 memory Effects 0.000 claims abstract description 325
- 230000002085 persistent effect Effects 0.000 claims abstract description 137
- 238000012545 processing Methods 0.000 claims description 40
- 230000007246 mechanism Effects 0.000 claims description 15
- 238000007639 printing Methods 0.000 claims description 9
- 230000000694 effects Effects 0.000 abstract description 3
- 239000013589 supplement Substances 0.000 abstract description 3
- 239000000047 product Substances 0.000 description 11
- 239000007787 solid Substances 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 230000001680 brushing effect Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000005055 memory storage Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011010 flushing procedure Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
Abstract
The invention provides a data storage method, which is used in a full flash device, wherein the full flash device comprises a persistent memory, a volatile memory and a hard disk; when the data volume is less than or equal to a first preset threshold value, starting the volatile memory to store the metadata to be processed; when the data volume is larger than a first preset threshold and smaller than or equal to a second preset threshold, simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed; and when the data volume is larger than the second preset threshold value, simultaneously starting the persistent memory, the volatile memory and the hard disk to store the metadata. The characteristics of the three storage modes are fully utilized, the volatile memory is preferentially adopted, when the metadata volume to be processed reaches a certain degree, the volatile memory and the persistent memory are simultaneously used, and a large-capacity hard disk is utilized as supplement of the scheme, so that the effect of considering both the storage and the use efficiency of the large-capacity metadata is realized.
Description
Technical Field
The invention relates to the technical field of storage, in particular to a data storage method, a data storage device, full-flash equipment and a computer readable storage medium.
Background
With the widespread use of information technology, storage technology is also constantly evolving. At present, the application of full flash memory products is increasingly widespread, and compared with the traditional memory products, a large amount of Metadata (Metadata) is used in the software architecture design of the full flash memory. Metadata is data describing objects such as information resources or data, and is generally stored in a storage service by using a memory, and problems caused by the fact that the use of the metadata in a full flash storage product is greatly increased are revealed.
The memory capacity in the storage device is limited, and for a full flash storage product, after the data amount of the metadata is increased, the allocation of the memory space becomes relatively difficult, and the memory in the general storage device may not meet the requirements of the current software design. At present, one solution is to increase the size of the memory of the storage device according to the requirement of the metadata memory, but due to the development of the related memory technology and the high price of the memory, the disadvantages caused by this method are also very obvious, that is, the cost of the device is greatly increased under the condition of limited memory space.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data storage method and apparatus, a full flash device, and a storage medium, so as to solve the problems of insufficient memory space and high cost of the full flash device in the prior art.
According to a first aspect, an embodiment of the present invention provides a data storage method, which is used in a full-flash device, where the full-flash device includes a persistent memory, a volatile memory, and a hard disk, and a capacity of the persistent memory is greater than a capacity of the volatile memory; the method comprises the following steps: acquiring the data volume of metadata to be processed; when the data volume is less than or equal to a first preset threshold value, starting the volatile memory to store the metadata to be processed; when the data volume is larger than a first preset threshold and smaller than or equal to a second preset threshold, simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed; and when the data volume is larger than the second preset threshold value, simultaneously starting the persistent memory, the volatile memory and the hard disk to store the metadata.
In the data storage method in this embodiment, the persistent memory, the volatile memory, and the hard disk are respectively arranged in the storage architecture of the full flash device, the characteristics of the three storage modes are fully utilized, and the volatile memory is preferentially adopted when less metadata needs to be processed, so as to ensure the storage operation efficiency; when the metadata amount to be processed reaches a certain degree, only the volatile memory is used when the occupancy rate of the volatile memory is too high or insufficient, the volatile memory and the persistent memory are used simultaneously, and the processing flow of the metadata is completed more efficiently by utilizing the characteristics of large capacity and relatively quick transmission speed of the persistent memory, so that the processing efficiency is considered on the premise that the metadata can be processed. When the processing capacity of the metadata reaches a higher value, persistent memory, volatile memory and hard disk storage are fully scheduled, a large-capacity hard disk is used as supplement of the scheme, and mass data storage is achieved.
With reference to the first aspect, in an optional implementation manner of the first aspect, the first preset threshold is set to 60% to 80% of the volatile memory capacity.
The first preset threshold is set to be 60% -80% of the capacity of the volatile memory, the occupancy rate of the volatile memory is guaranteed while the data are efficiently stored through the volatile memory, and when the data volume is too large, a mode that the volatile memory and the persistent memory are jointly started is started.
With reference to the first aspect, in an optional implementation manner of the first aspect, the second preset threshold is set to 60% to 80% of the persistent memory capacity. At this time, the amount of metadata to be processed is already large, the usage rates of the persistent memory and the volatile memory are high, and the processing efficiency is affected.
With reference to the first aspect, in an optional implementation manner of the first aspect, the simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed when the data amount is greater than a first preset threshold and less than or equal to a second preset threshold includes: setting a volatile memory as a cache of the persistent memory; acquiring the use frequency of the metadata; and storing the metadata of which the usage frequency domain is higher than a first frequency threshold value in a volatile memory, and printing the rest metadata down to the persistent memory.
In this way, by setting the volatile memory as the cache of the persistent memory, the relatively hot data with high use frequency is stored in the volatile memory, so that the data with more use can be read more quickly, and the rest of relatively cold data is stored in the persistent memory, and is stored in the persistent memory in a differentiated manner according to the cold and hot degrees of the data, so that the processing efficiency can be further improved.
With reference to the first aspect, in an optional implementation manner of the first aspect, the simultaneously starting the persistent memory, the volatile memory, and the hard disk to store the metadata when the data amount is greater than the second preset threshold includes: acquiring the use frequency of the metadata; storing the metadata with the usage frequency domain higher than or equal to a first frequency threshold in a volatile memory; printing down metadata with the usage frequency below the first frequency threshold and above or equal to a second frequency threshold into persistent memory; and printing the metadata with the use frequency lower than the second frequency threshold value into the hard disk.
In this mode, when the data amount is very large, the data is stored in a sorted manner according to the use degree of the data, and the data with the least use is printed on the hard disk.
With reference to the first aspect, in an optional implementation manner of the first aspect, the method further includes, when a failure occurs, storing the metadata in the volatile memory in a persistent memory through a power down protection mechanism.
In the method, the metadata in the volatile memory is stored in the persistent memory when the fault occurs, so that the safety of the data is further improved, the effective data can not be volatilized due to the fault, and conditions are created for normal processing of the data after the subsequent fault is recovered.
With reference to the first aspect, in an optional implementation manner of the first aspect, the method further includes, when the failure is recovered, recovering the metadata, which was in the persistent memory when the failure occurred, to the volatile memory again. In the mode, the effective data in the original volatile memory is restored after the fault is recovered, so that the data can be efficiently processed, and the operation efficiency of the data is guaranteed.
According to a second aspect, an embodiment of the present invention provides a data storage apparatus, which is used in an all-flash device, where the all-flash device includes a persistent memory, a volatile memory, and a hard disk, and a capacity of the persistent memory is greater than a capacity of the volatile memory; the device includes:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring the data volume of metadata to be processed;
the first processing unit is used for starting the volatile memory to store the metadata to be processed when the data volume is less than or equal to a first preset threshold value;
the second processing unit is used for simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed when the data volume is larger than a first preset threshold and smaller than or equal to a second preset threshold;
and the third processing unit is used for simultaneously starting the persistent memory, the volatile memory and the hard disk storage when the data volume is larger than the second preset threshold.
In an alternative embodiment according to the second aspect, the first predetermined threshold is set to 60% -80% of the volatile memory capacity.
In an alternative embodiment according to the second aspect, the second preset threshold is set to 60% -80% of the persistent memory capacity.
According to a second aspect, in an optional implementation of the second aspect, the second processing unit comprises:
the cache subunit is used for setting the volatile memory as the cache of the persistent memory;
a first frequency acquiring subunit configured to acquire a frequency of use of the metadata;
the first processing subunit is configured to store the metadata with the usage frequency domain higher than the first frequency threshold in a volatile memory, and to print the rest of the metadata down to the persistent memory.
According to a second aspect, in an optional implementation of the second aspect, the third processing unit comprises:
a second frequency acquiring subunit configured to acquire a frequency of use of the metadata;
a second storage subunit, configured to store, in a volatile memory, the metadata of which the usage frequency domain is higher than or equal to a first frequency threshold;
a third storage subunit, configured to print the metadata with the usage frequency lower than the first frequency threshold and higher than or equal to a second frequency threshold into a persistent memory;
a fourth storage subunit, configured to print the metadata with the usage frequency lower than the second frequency threshold value to the hard disk.
According to the second aspect, in an optional implementation manner of the second aspect, the apparatus further includes a failure processing unit, configured to, when a failure occurs, store the metadata in the volatile memory in the persistent memory through a power down protection mechanism.
According to the second aspect, in an optional implementation manner of the second aspect, the apparatus further includes a failure recovery unit, configured to recover, when the failure is recovered, the metadata that was in the persistent memory when the failure occurred, to the volatile memory again.
According to a third aspect, an embodiment of the present invention provides an all-flash device, including a storage and a processor, where the storage includes a persistent memory, a volatile memory, and a hard disk, and the storage and the processor are communicatively connected to each other, the storage stores computer instructions, and the processor executes the computer instructions, thereby executing the data storage method in the first aspect or any one of the implementation manners of the first aspect.
According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions for causing a computer to execute the data storage method in the first aspect or any one implementation manner of the first aspect.
According to a fifth aspect, an embodiment of the present invention provides a computer program product, the computer program product comprising a computer program stored on a computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the data storage method of the first aspect or any one of the embodiments of the first aspect.
Drawings
The features and advantages of the present invention will be more clearly understood by reference to the accompanying drawings, which are illustrative and not to be construed as limiting the invention in any way, and in which:
FIG. 1 is a flow diagram that illustrates the steps of a data storage method in one embodiment;
FIG. 2 is a flow chart showing the steps of a data storage method in another embodiment;
FIG. 3 is a flow chart illustrating the steps of a data storage method in another embodiment;
FIG. 4 is a flow chart illustrating the steps of a data storage method in another embodiment;
FIG. 5 is a block diagram showing the structure of a data storage device in one embodiment;
FIG. 6 is a block diagram showing the construction of a data storage device in another embodiment;
FIG. 7 shows a block diagram of a full flash device of an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the data storage method provided in the embodiment of the present application is used in a full flash device, and may be implemented in a software, hardware, or a combination of the software and the hardware, where the full flash device (also referred to as a full flash array) refers to an independent storage array or device completely composed of solid state storage media, and a Solid State Disk (SSD) or other flash storage media is used in a storage subsystem to replace a conventional hard disk (HDD). Typically, a full flash array can provide 50 to 100 million IOPS (input/output per second) in a single script with latency under 1 millisecond. The relatively expensive SSD of solid-state storage arrays makes full flash memory arrays suitable only for high IOPS applications, but due to their advantages in high performance, flash memory media are widely valued and used despite their high price.
The application of the full flash memory product is increasingly wide, compared with the traditional memory product, a large amount of metadata is used in the software architecture design of the full flash memory, and the metadata is usually stored by using a memory in a storage service. Meanwhile, since metadata is used as data describing data, part of metadata content is very important, and there is a certain risk in using a volatile memory medium for storage.
In order to solve the problem that a general memory cannot meet the requirement of metadata processing capacity in the full-flash device, this embodiment provides a data storage method, which is used in the full-flash device, where the full-flash device includes a persistent memory, a volatile memory, and a hard disk, and stores metadata by using the persistent memory, the volatile memory, and the hard disk in different situations, and increases a capacity judgment mechanism and a metadata storage process, thereby achieving an effect of considering both the storage and the usage efficiency of the metadata with a large capacity.
The persistent memory may be an intel loft nonvolatile memory (Optane DIMM), and the volatile memory refers to a conventional memory such as a DRAM. The hard disk refers to a Solid State Disk (SSD) or other flash memory medium.
The method is characterized in that a persistent memory intel Optane DC is applied in the full flash memory device and is used as a storage space of important metadata, compared with a commonly used volatile memory (such as a DRAM), the persistent memory has larger capacity, for example, the intel Optane DC mainly has 128GB, 256GB and 512GB centralized types, the space capacity is greatly improved compared with that of a common memory, and the problem of insufficient memory space caused by larger metadata data volume in the full flash memory can be effectively solved.
Compared with the common memory, the other important advantage of the persistent memory is embodied in price, the persistent memory has lower cost, and the cost of the product can be saved while the memory space is improved; meanwhile, the persistent memory also has higher performance, and with the further development of the technology, the performance of the persistent memory is correspondingly improved.
Compared with a common memory, the larger difference of the persistent memory is that the persistent memory can have nonvolatile property, and when the persistent memory is set to be in an App Direct working mode, the persistent memory has the characteristic that power-down data is not lost, so that important metadata content stored in the persistent memory has a larger advantage in read-write performance compared with a common Solid State Disk (SSD).
In the full flash device in this embodiment, the capacity of the persistent memory is greater than the capacity of the volatile memory, and the capacity of the general persistent memory is in specifications of 128G, 256G, 512G, and the like, which can reach about 3T, and the specification of the volatile memory is in specifications of 8G, 32G, 64G, and the like. The capacity of the persistent memory and the volatile memory is reasonably determined according to the capacity of a hard disk of the full-flash device, and the capacity of the persistent memory and the volatile memory can be generally set to be more than 8 times that of the volatile memory.
In the overall architecture design, the persistent memory is independently arranged as an intermediate layer between the original (DRAM) and the rear-end hard disk, a volatile memory-persistent memory-hard disk three-layer structure is generally formed, and the full flash memory metadata refreshing flow correspondingly changes based on the change of the architecture. Considering that a judgment mechanism for the metadata amount is added, and the metadata amount and the sizes of the storage spaces of the volatile memory and the persistent memory in the current device are integrated, as shown in fig. 1, the data storage method in this embodiment includes the following steps:
and S1, acquiring the data volume of the metadata to be processed.
Metadata is data that describes an object such as an information resource or data, and is used for the purpose of: identifying a resource; evaluating the resources; tracking changes of the resource in the using process; the method realizes simple and efficient management of a large amount of networked data; the information resources are effectively discovered, searched and integrally organized, and the used resources are effectively managed.
In a full flash device, the metadata processing amount is greatly increased. In order to reasonably allocate storage resources according to the size of the data volume in the scheme, the data volume of the metadata to be processed needs to be acquired at first, and the data volume at the position can be directly acquired through the metadata.
And S2, when the data volume is less than or equal to a first preset threshold value, starting the volatile memory to store the metadata to be processed.
After the data volume of the metadata to be processed is obtained, whether the data volume is less than or equal to a first preset threshold is judged, wherein the first preset threshold is set to be 60% -80% of the capacity of the volatile memory, for example, values such as 60%, 70%, 75% or 80% can be selected, and the first preset threshold is reasonably set as required.
The first preset threshold is set to be 60% -80% of the capacity of the volatile memory, the high-efficiency storage of data through the volatile memory is guaranteed, meanwhile, the occupancy rate of the volatile memory is not too high, and when the data volume is too large, a mode that the volatile memory and the persistent memory are jointly started is started.
And S3, when the data volume is larger than a first preset threshold value and smaller than or equal to a second preset threshold value, simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed.
When the data amount is greater than the first preset threshold, further determining whether the data amount is less than a second preset threshold, where the second preset threshold is set to be 60% to 80% of the persistent memory capacity, and may be selected as a value such as 60%, 70%, 75%, or 80% of the persistent memory capacity, and is reasonably set as needed. When the metadata amount is less than or equal to a second preset threshold, the metadata amount needing to be processed is large, a high proportion of space of the volatile memory is occupied or the capacity of the volatile memory is exceeded, at this time, the persistent memory is started, the volatile memory DRAM is used as a cache of the persistent memory, data with higher use frequency is stored in the DRAM, and other parts are printed in the persistent memory.
After the persistent memory is introduced, the storage of the metadata is stored in the volatile memory DRAM and the persistent memory through the memory bus, and the volatile memory DRAM is used as the cache of the persistent memory in the storage architecture at this time. The method applies the persistent memory in the full flash memory storage product, utilizes the relevant interface of the persistent memory and combines a memory cold and hot data judgment mechanism to store the metadata stored in the volatile memory DRAM in the persistent memory relatively cold data, utilizes the characteristics of large capacity and relatively high transmission speed of the persistent memory to more efficiently finish the processing flow of the metadata, and solves the problem of insufficient memory space of the full flash memory storage product. Meanwhile, due to the non-volatility of the persistent memory, the metadata still does not need to be written in the rear-end SSD solid state disk, and the storage operation efficiency of the data is guaranteed.
And S4, when the data volume is larger than the second preset threshold, simultaneously starting the persistent memory, the volatile memory and the hard disk to store the metadata.
When the data volume of the metadata is larger than the second threshold, the utilization rate of the persistent memory and the volatile memory is high, the processing efficiency is affected by increasing the data volume, and at the moment, the volatile memory, the persistent memory and the rear-end hard disk are started to jointly store the data, so that the efficiency is considered while the data processing is ensured.
The storage capacity of metadata in a full flash memory storage system is also increasingly demanded, and under the current design architecture, when the amount of metadata is very large and exceeds more than 60% -80% of the space of a persistent memory, the least frequently used metadata is flushed and written into a back-end hard disk (SSD) through the persistent memory in a down-flushing mode so as to solve the problem of insufficient memory space, so that the problem of low efficiency exists.
And when the metadata in the persistent memory reaches a certain amount, writing the metadata content in part of the persistent memory into the SSD disk in a refreshing mode. And storing the metadata in a volatile memory, a persistent memory and a hard disk respectively according to the heat of the data.
In the current full flash memory software design, the problem that a large amount of metadata cannot be stored can be solved on the premise of not obviously increasing the product cost. Because the data are stored in the volatile memory, the persistent memory and the hard disk in sequence according to the heat, the data with higher use frequency are preferentially stored in the volatile memory, so that the operation efficiency of the data can be guaranteed as much as possible during the operation of the data, and the effect of taking the storage and the use efficiency of the large-capacity metadata into consideration is realized by increasing a capacity judgment mechanism and a metadata storage process.
In the data storage method in this embodiment, the persistent memory, the volatile memory, and the hard disk are respectively arranged in the storage architecture of the full flash device, the characteristics of the three storage modes are fully utilized, and the volatile memory is preferentially adopted when less metadata needs to be processed, so as to ensure the storage operation efficiency; when the metadata amount to be processed reaches a certain degree, only the volatile memory is used when the occupancy rate of the volatile memory is too high or insufficient, the volatile memory and the persistent memory are used simultaneously, and the processing flow of the metadata is completed more efficiently by utilizing the characteristics of large capacity and relatively quick transmission speed of the persistent memory, so that the processing efficiency is considered on the premise that the metadata can be processed. When the processing capacity of the metadata reaches a higher value, persistent memory, volatile memory and hard disk storage are fully scheduled, a large-capacity hard disk is used as supplement of the scheme, and mass data storage is achieved.
In a specific embodiment, as shown in fig. 2, the step S3 includes:
s31, setting a volatile memory as a cache of the persistent memory;
s32, acquiring the use frequency of the metadata; the usage frequency may be used to represent the heat of the metadata.
And S33, storing the metadata of which the usage frequency domain is higher than the first frequency threshold in a volatile memory, and brushing the rest metadata in the persistent memory. The first frequency threshold value can be set according to different occasions and needs. The average usage frequency may be calculated by counting the usage frequencies of all metadata of the metadata, and the average usage frequency is used as the first frequency threshold. Of course, the first frequency threshold may be set as a percentage of the average frequency of use, and may be set as appropriate.
In this way, by setting the volatile memory as the cache of the persistent memory, the relatively hot data with high use frequency is stored in the volatile memory, so that the data with more use can be read more quickly, and the rest of relatively cold data is stored in the persistent memory, and is stored in the persistent memory in a differentiated manner according to the cold and hot degrees of the data, so that the processing efficiency can be further improved.
As a specific implementation manner, step S4 is as shown in fig. 3, and further includes:
and S41, acquiring the use frequency of the metadata. The frequency of use identifies the heat of the metadata.
And S42, storing the metadata of which the usage frequency domain is higher than or equal to the first frequency threshold in a volatile memory.
S43, brushing the metadata of which the use frequency is lower than the first frequency threshold and higher than or equal to a second frequency threshold into a persistent memory.
Here, the first frequency threshold and the second frequency threshold are similar to those in step S3, and may be set according to the percentage of the average usage frequency, for example, the first frequency threshold is set as the average usage frequency, and the second frequency threshold is set as half of the average usage frequency, and is set appropriately according to the needs. Or may be set to a specific value under a specific scene.
And S44, brushing the metadata with the use frequency lower than the second frequency threshold value into the hard disk.
In this mode, when the data amount is very large, the data is stored in a sorted manner according to the use degree of the data, and the data with the least use is printed on the hard disk.
As a further implementation manner, in order to cope with a system failure, the data storage method further includes:
and when a fault occurs, storing the metadata in the volatile memory in a persistent memory through a power failure protection mechanism. When a storage controller has a fault scene, the metadata in the DRAM is also quickly stored in the persistent memory through a power-down protection mechanism.
In the method, the metadata in the volatile memory is stored in the persistent memory when the fault occurs, so that the safety of the data is further improved, the effective data can not be volatilized due to the fault, and conditions are created for normal processing of the data after the subsequent fault is recovered.
Further comprising restoring the metadata in the persistent memory that existed at the time of the failure to the volatile memory again when the failure is restored. In the mode, the effective data in the original volatile memory is restored after the fault is recovered, so that the data can be efficiently processed, and the operation efficiency of the data is guaranteed. When the fault controller is recovered, partial data in the persistent memory can be recovered to the volatile memory DRAM according to a cold and hot data mechanism, and the rest of the data are transmitted and used through the DRAM-persistent memory architecture in the full flash device according to specific service requirements.
Therefore, under the scenes of normal use, transmission process and fault recovery of metadata, the mechanisms of pre-reading, writing, cache hitting and the like are also applied to the persistent memory, namely, the mechanism contained in the DRAM layer is also existed after the flash memory is written into the persistent memory, so that the mechanism for improving the data transmission efficiency is also integrated into the memories of different types of the upper layer and the lower layer, and the further improvement of the metadata transmission efficiency is realized.
In order to better understand the data storage method provided in the embodiment of the present application, an embodiment of the present application provides an overall flowchart of the data storage method, and the overall flowchart is used in a full-flash device, where the full-flash device includes a persistent memory, a volatile memory, and a hard disk, and a capacity of the persistent memory is greater than a capacity of the volatile memory, as shown in fig. 4, the method specifically includes the following steps:
s401, acquiring the data volume of metadata to be processed;
s402, when the data volume is smaller than or equal to a first preset threshold value, starting the volatile memory to store the metadata to be processed; the first preset threshold is set to be 60% -80% of the volatile memory capacity; the method comprises the following steps:
setting a volatile memory as a cache of the persistent memory;
acquiring the use frequency of the metadata;
and storing the metadata of which the usage frequency domain is higher than a first frequency threshold value in a volatile memory, and printing the rest metadata down to the persistent memory.
S403, when the data volume is larger than a first preset threshold and smaller than or equal to a second preset threshold, simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed; the second preset threshold is set to be 60% -80% of the persistent memory capacity; the method comprises the following steps:
acquiring the use frequency of the metadata;
storing the metadata with the usage frequency domain higher than or equal to a first frequency threshold in a volatile memory;
printing down metadata with the usage frequency below the first frequency threshold and above or equal to a second frequency threshold into persistent memory;
and printing the metadata with the use frequency lower than the second frequency threshold value into the hard disk.
S404: and when the data volume is larger than the second preset threshold value, simultaneously starting the persistent memory, the volatile memory and the hard disk to store the metadata.
S405, when a fault occurs, storing the metadata in the volatile memory in a persistent memory through a power failure protection mechanism.
S406, when the recovery is carried out after the failure, the metadata in the persistent memory when the failure occurs is recovered to the volatile memory again.
It should be understood that although the various steps in the flow charts of fig. 1-4 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-4 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
Correspondingly, referring to fig. 5, an embodiment of the present invention provides a data storage apparatus, which is used in a full-flash device, where the full-flash device includes a persistent memory, a volatile memory, and a hard disk, and a capacity of the persistent memory is greater than a capacity of the volatile memory; as shown in fig. 5, the apparatus includes:
an obtaining unit 01, configured to obtain a data amount of metadata to be processed;
the first processing unit 02 is configured to start the volatile memory to store the metadata to be processed when the data amount is less than or equal to a first preset threshold;
the second processing unit 03 is configured to, when the data amount is greater than a first preset threshold and less than or equal to a second preset threshold, simultaneously start the persistent memory and the volatile memory to store the metadata to be processed;
and the third processing unit 04 is configured to start the persistent memory, the volatile memory, and the hard disk storage simultaneously when the data amount is greater than the second preset threshold.
Wherein the first preset threshold is set to 60% -80% of the volatile memory capacity. The second preset threshold is set to 60% -80% of the persistent memory capacity.
As a specific implementation, the second processing unit 03 includes:
the cache subunit is used for setting the volatile memory as the cache of the persistent memory;
a first frequency acquiring subunit configured to acquire a frequency of use of the metadata;
the first processing subunit is configured to store the metadata with the usage frequency domain higher than the first frequency threshold in a volatile memory, and to print the rest of the metadata down to the persistent memory.
As a specific embodiment, the third processing unit 04 includes:
a second frequency acquiring subunit configured to acquire a frequency of use of the metadata;
a second storage subunit, configured to store, in a volatile memory, the metadata of which the usage frequency domain is higher than or equal to a first frequency threshold;
a third storage subunit, configured to print the metadata with the usage frequency lower than the first frequency threshold and higher than or equal to a second frequency threshold into a persistent memory;
a fourth storage subunit, configured to print the metadata with the usage frequency lower than the second frequency threshold value to the hard disk.
As a further embodiment, as shown in fig. 6, the apparatus further includes a failure processing unit 05, configured to, when a failure occurs, store the metadata in the volatile memory in the persistent memory through a power down protection mechanism.
Further, the system comprises a failure recovery unit 06 for recovering the metadata in the persistent memory existing at the time of failure to the volatile memory again when the failure is recovered.
For specific limitations and beneficial effects of the data storage device, reference may be made to the above limitations of the data storage method, which are not described herein again. The modules in the battery performance testing device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Fig. 7 is a schematic diagram of a hardware structure of a full-flash device according to an embodiment of the present invention, as shown in fig. 7, the device includes one or more processors 710 and a storage 720, where the storage 720 includes a persistent memory, a volatile memory, and a hard disk, and one processor 710 is taken as an example in fig. 7. The apparatus may further include: an input device 730 and an output device 740.
The processor 710, the memory 720, the input device 730, and the output device 740 may be connected by a bus or other means, such as the bus connection in fig. 7.
The memory 720, serving as a non-transitory computer-readable storage medium, includes a persistent memory, a volatile memory, and a hard disk, and can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the service management method in the embodiment of the present application. The processor 710 implements the above-described data storage method by executing non-transitory software programs, instructions, and modules stored in the memory 720 to perform various functional applications of the server and data processing.
The memory 720 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data used as needed or desired, and the like. Further, the memory 720 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 720 optionally includes memory located remotely from processor 710, which may be connected to a data processing device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 730 may receive input numeric or character information and generate key signal inputs related to user settings and function control. The output device 740 may include a display device such as a display screen.
The one or more modules are stored in the memory 720 and, when executed by the one or more processors 710, perform the methods shown in fig. 1-4.
The product can execute the method provided by the embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. Details of the technique not described in detail in the present embodiment may be specifically referred to the related description in the embodiments shown in fig. 1 to 4.
The embodiment of the invention also provides a non-transitory computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the service management method in any method embodiment. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A data storage method is characterized in that the method is used in a full flash device, the full flash device comprises a persistent memory, a volatile memory and a hard disk, and the capacity of the persistent memory is larger than that of the volatile memory; the method comprises the following steps:
acquiring the data volume of metadata to be processed;
when the data volume is less than or equal to a first preset threshold value, starting the volatile memory to store the metadata to be processed;
when the data volume is larger than a first preset threshold and smaller than or equal to a second preset threshold, simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed;
and when the data volume is larger than the second preset threshold value, simultaneously starting the persistent memory, the volatile memory and the hard disk to store the metadata.
2. The method according to claim 1, wherein the first predetermined threshold is set to 60% -80% of the volatile memory capacity.
3. The method according to claim 1 or 2, wherein the second preset threshold is set to 60% -80% of the persistent memory capacity.
4. The method according to claim 3, wherein when the data amount is greater than a first preset threshold and less than or equal to a second preset threshold, simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed comprises:
setting a volatile memory as a cache of the persistent memory;
acquiring the use frequency of the metadata;
and storing the metadata of which the usage frequency domain is higher than a first frequency threshold value in a volatile memory, and printing the rest metadata down to the persistent memory.
5. The method of claim 3, wherein said concurrently starting the persistent memory, volatile memory, and hard disk to store the metadata when the amount of data is greater than the second preset threshold comprises:
acquiring the use frequency of the metadata;
storing the metadata with the usage frequency domain higher than or equal to a first frequency threshold in a volatile memory;
printing down metadata with the usage frequency below the first frequency threshold and above or equal to a second frequency threshold into persistent memory;
and printing the metadata with the use frequency lower than the second frequency threshold value into the hard disk.
6. The method of claim 4 or 5, further comprising saving the metadata in the volatile memory in persistent memory through a power down protection mechanism when a failure occurs.
7. The method of claim 6, further comprising restoring metadata in the persistent memory that existed at the time of the failure to the volatile memory again when the failure was recovered.
8. A data storage device is used in a full flash device, wherein the full flash device comprises a persistent memory, a volatile memory and a hard disk, and the capacity of the persistent memory is greater than that of the volatile memory; the device includes:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring the data volume of metadata to be processed;
the first processing unit is used for starting the volatile memory to store the metadata to be processed when the data volume is less than or equal to a first preset threshold value;
the second processing unit is used for simultaneously starting the persistent memory and the volatile memory to store the metadata to be processed when the data volume is larger than a first preset threshold and smaller than or equal to a second preset threshold;
and the third processing unit is used for simultaneously starting the persistent memory, the volatile memory and the hard disk storage when the data volume is larger than the second preset threshold.
9. An all-flash device, comprising a storage and a processor, wherein the storage comprises a persistent memory, a volatile memory and a hard disk, the storage and the processor are communicatively connected, the storage stores computer instructions, and the processor executes the computer instructions to perform the data storage method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing computer instructions for causing a computer to perform the data storage method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111124219.9A CN113835636A (en) | 2021-09-24 | 2021-09-24 | Data storage method and device, full-flash equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111124219.9A CN113835636A (en) | 2021-09-24 | 2021-09-24 | Data storage method and device, full-flash equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113835636A true CN113835636A (en) | 2021-12-24 |
Family
ID=78969991
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111124219.9A Withdrawn CN113835636A (en) | 2021-09-24 | 2021-09-24 | Data storage method and device, full-flash equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113835636A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114218017A (en) * | 2022-02-18 | 2022-03-22 | 苏州浪潮智能科技有限公司 | Data recovery method, system and device and storage equipment |
-
2021
- 2021-09-24 CN CN202111124219.9A patent/CN113835636A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114218017A (en) * | 2022-02-18 | 2022-03-22 | 苏州浪潮智能科技有限公司 | Data recovery method, system and device and storage equipment |
| CN114218017B (en) * | 2022-02-18 | 2022-05-10 | 苏州浪潮智能科技有限公司 | Data recovery method, system and device and storage equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10261853B1 (en) | Dynamic replication error retry and recovery | |
| CN102549524B (en) | Adaptive power conservation in storage clusters | |
| US9110669B2 (en) | Power management of a storage device including multiple processing cores | |
| EP3070591A1 (en) | Cold and hot data identification threshold calculation method, apparatus and system | |
| US20140006687A1 (en) | Data Cache Apparatus, Data Storage System and Method | |
| CN109857239B (en) | Storage control method and device | |
| CN108205473B (en) | Memory processing method and device, computer device and computer readable storage medium | |
| CN108829510B (en) | Thread binding processing method and device | |
| CN103218274A (en) | Failure accumulation preventing method and solid state disk | |
| CN108089825B (en) | Storage system based on distributed cluster | |
| TWI732791B (en) | Data storage device and operating method thereof | |
| CN113835636A (en) | Data storage method and device, full-flash equipment and computer readable storage medium | |
| CN108008917A (en) | Storage device and the method for controlling its linking status | |
| CN108205501B (en) | Memory recovery method and device, computer device and computer readable storage medium | |
| CN109683983B (en) | Method and equipment for generating and loading mirror image file | |
| WO2017147794A1 (en) | Differential data backup method and device | |
| CN112948169A (en) | Data backup method, device, equipment and storage medium | |
| CN106598768A (en) | Write request processing method, device and data center | |
| CN108616598A (en) | Method of data synchronization, device and distributed memory system | |
| CN110018797B (en) | Data migration method, device and equipment and readable storage medium | |
| CN113377278A (en) | Solid state disk, garbage recycling and controlling method, equipment, system and storage medium | |
| CN109960474A (en) | Data-updating method, device, equipment and medium based on automatic simplify configuration | |
| CN112035070B (en) | Dirty data refreshing method, device and equipment and computer readable storage medium | |
| WO2022166265A1 (en) | Data recovery method and apparatus, device and medium | |
| CN109491593B (en) | Data storage management system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20211224 |