CN112486423A - Distributed storage method and device - Google Patents
Distributed storage method and device Download PDFInfo
- Publication number
- CN112486423A CN112486423A CN202110158659.XA CN202110158659A CN112486423A CN 112486423 A CN112486423 A CN 112486423A CN 202110158659 A CN202110158659 A CN 202110158659A CN 112486423 A CN112486423 A CN 112486423A
- Authority
- CN
- China
- Prior art keywords
- value
- data block
- data
- medium
- storage medium
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention provides a distributed storage method and a distributed storage device, and relates to the technical field of computers. The method comprises the following steps: acquiring the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium; comparing the medium capacity with the data value to obtain a comparison result; and carrying out storage management on the data blocks according to the comparison result. And comparing the medium capacity and the data value to judge whether the benefit maximization is realized when the data block is stored in the storage medium at the current moment. By comparing the medium capacity of the storage medium with the data value of the data block, the storage and the flow of the data block in the distributed storage are realized to achieve higher efficiency and accuracy, the data block is moved to the place where the data block should be moved most necessarily, the medium benefit is maximized, and therefore accurate and efficient flow storage is realized.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a distributed storage method and a distributed storage device.
Background
Currently, distributed storage, although being manufactured by a large number of vendors, uses a data storage flow principle including three major categories, i.e., a storage medium flow type, a storage pool flow type, and a storage copy flow type. The storage medium flow type is to bind a medium with a higher grade to a medium with a lower grade by using software or an algorithm to form a mixed medium, then store data in the virtual mixed medium, and perform more efficient flow and exchange in two different physical media by using a fixed algorithm, but the storage mode has the defect of inconvenient expansion management. The storage pool streaming type is to stream and exchange data between different storage pools, and different storage pools generally use different storage media, so that the convenience of expanding the media is improved to some extent. The storage copy flow type is to allow multiple copies of the distributed storage to be stored in different media, for example, the distributed storage adopts a three-copy protection mode, the system stores the first two copies in a high-level medium, stores the last copy in a lower-level medium, and returns the first two copies to the client after the writing is completed. The method solves the problem of fixing the medium model to a great extent, basically can realize the expansion of the medium without correlation, but has larger performance influence when the medium with lower performance can not bear under the condition of slightly higher performance requirement; in addition, the safety of data is sacrificed to a certain extent; most critically, this approach does not yield high gains in available capacity. The main purpose of using different media in the same storage is to reduce costs and increase efficiency, since different data also have different performance requirements on the media. The three traditional modes have a serious defect that the storage heterogeneous capacity is poor, that is, the number and the capacity of different media are required to be basically consistent, the types of the media can only support two types, however, the media in the market are updated very fast, so that the media hardware which is the same as the media hardware before is difficult to purchase after capacity expansion occurs, and the maintenance difficulty and the cost are high.
Disclosure of Invention
The invention aims to provide a distributed storage method and a distributed storage device, which are used for solving the problem of poor storage heterogeneous capability in the prior art.
In a first aspect, an embodiment of the present application provides a distributed storage method, where the method includes:
acquiring the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium;
comparing the medium capacity with the data value to obtain a comparison result;
and carrying out storage management on the data blocks according to the comparison result.
In the implementation process, the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium are firstly obtained, and the medium capability and the data value are compared to judge whether the benefit maximization is realized when the data block is stored in the storage medium at the current moment. And then, carrying out storage management on the data block according to a comparison result, if the medium capacity is not matched with the data value, allocating a proper storage medium for the data block again for storage, namely, comparing the medium capacity of the storage medium with the data value of the data block, and finally achieving higher efficiency and accuracy of storage and flow of the data block in distributed storage, so that the data block can go to a place where the data block should go most when necessary, thereby realizing medium benefit maximization and realizing accurate and efficient flow storage.
In some embodiments of the present invention, the step of obtaining the media capability of the storage medium at the current time comprises:
acquiring attribute parameters, health degree, capacity space and busyness degree of a storage medium;
and weighting the attribute parameters, the health degree, the capacity space and the busy degree respectively to calculate the medium capacity of the storage medium at the current moment.
In the above embodiment, the medium capacity evaluation value is weighted and calculated according to the inherent property capacity, health degree, capacity space and busyness degree of the medium, so that the performance variation of the storage medium can be grasped in real time, and the medium capacity of the storage medium can be accurately acquired.
In some embodiments of the invention, the step of obtaining a data value of a data block stored in a storage medium comprises:
obtaining an I/O model value influence value, a service importance value influence value and a read-write hit rate value influence value of a data block;
and calculating the data value of the data block according to the data block value quantity calculation model, the I/O model value influence value of the data block, the service importance value influence value and the read-write hit rate value influence value.
In some embodiments of the invention, the data acquisition module comprises:
an I/O model value influence value acquisition unit for acquiring an I/O model value influence value according to a first relationship function:
obtaining constant parameters specified when allocating storage space
As a business importance value impact value; wherein the content of the first and second substances,
；
and obtaining a value influence value of the read-write hit rate according to the following second relation function:
calculating a model according to the value of the data block, and calculating the value influence value of the I/O model and the service importance value influence value of the data block
And read-write hit rate value impact value
Calculating the data value of the data block:
Through the process, a data block value calculation model is comprehensively constructed according to three indexes of an I/O model value influence value, a business importance value influence value and a read-write hit rate value influence value of the data block, and the data value can be objectively and comprehensively evaluated.
In some embodiments of the present invention, the step of performing storage management on the data block according to the comparison result includes:
if the comparison result shows that the data value is reduced, storing the data block into a new storage medium with the medium capability lower than that of the current storage medium;
and if the comparison result shows that the data value is increased, recording the liability condition of the current storage medium, and carrying out storage management on the data block according to the liability condition.
In some embodiments of the present invention, the step of recording the liability condition of the current storage medium, and performing storage management on the data block according to the liability condition comprises:
when the response of the current storage medium is not timely in the process of operating the data block, increasing the liability value of the current storage medium;
and when the liability value is larger than a preset threshold value, storing the data block into a new storage medium with the medium capability higher than that of the current storage medium.
In the implementation process, the data blocks in the storage medium can be flowed to a lower-level storage medium if the data blocks in the storage medium are depreciated, but if the data blocks in the storage medium obtain more reading and writing frequencies, the heat is high, and the data value is high, according to the logic of maximizing the benefit of the storage medium, the storage medium still catches the data blocks to enlarge the benefit of the storage medium, but if the data blocks do not respond in time in the value-added process, the storage medium is charged accordingly, and in order to eliminate the charge, the storage medium needs to transfer the high-value data blocks to other high-level storage media to eliminate the charge of the storage medium, so that the problem that the storage medium cannot bear the reading and writing operations of the data blocks with high data values is effectively avoided, and the high efficiency of data storage is further ensured.
In some embodiments of the invention, the method further comprises:
acquiring data values of all data blocks to be written, and arranging all the data blocks from high to low according to the data value corresponding to each data block to generate a data block queue to be written;
acquiring the medium capacity of all storage media, and arranging all the storage media from high to low according to the medium capacity corresponding to each storage medium to generate a medium queue;
writing the data block with the highest data value in the data block queue into the storage medium with the highest capacity in the medium queue;
deleting the written data blocks from the data block queue to be written, and transferring to the step of acquiring the medium capabilities of all the storage media, and arranging all the storage media from high to low according to the medium capabilities corresponding to each storage medium to generate a medium queue until no data block to be written exists in the data block queue to be written.
In a second aspect, an embodiment of the present application provides a distributed storage apparatus, including:
the data acquisition module is used for acquiring the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium;
the comparison module is used for comparing the medium capacity and the data value to obtain a comparison result;
and the storage management module is used for carrying out storage management on the data blocks according to the comparison result.
In some embodiments of the invention, the data acquisition module comprises:
a storage medium parameter acquiring unit for acquiring attribute parameters, health degree, capacity space and busy degree of the storage medium;
and the medium capacity calculating unit is used for weighting the attribute parameters, the health degree, the capacity space and the busy degree respectively so as to calculate and obtain the medium capacity of the storage medium at the current moment.
In some embodiments of the invention, the data acquisition module comprises:
the data block parameter acquisition unit is used for acquiring an I/O model value influence value, a service importance value influence value and a read-write hit rate value influence value of the data block;
and the data value calculation unit is used for calculating the data value of the data block according to the data block value quantity calculation model, the I/O model value influence value of the data block, the service importance value influence value and the read-write hit rate value influence value.
In some embodiments of the invention, the data acquisition module comprises:
an I/O model value influence value acquisition unit for acquiring an I/O model value influence value according to a first relationship function:
a business importance value influence value acquisition unit for acquiring constant parameters specified when allocating storage space
As a business importance value impact value; wherein the content of the first and second substances,
；
a reading and writing hit rate value influence value obtaining unit, configured to obtain a reading and writing hit rate value influence value according to the following second relation function:
a data value acquisition unit for calculating the value influence value of the I/O model and the service importance value influence value of the model and the data block according to the value of the data block
And calculating the data value of the data block according to the read-write hit rate value influence value:
In some embodiments of the invention, the storage management module comprises:
the first storage management unit is used for storing the data blocks into a new storage medium with the medium capability lower than that of the current storage medium if the comparison result shows that the data value is reduced;
and the second storage management unit is used for recording the liability condition of the current storage medium and carrying out storage management on the data block according to the liability condition if the comparison result shows that the data value is increased.
In some embodiments of the invention, the second storage management unit comprises:
the liability value increasing subunit is used for increasing the liability value of the current storage medium when the response of the current storage medium is not timely in the process of operating the data block;
and the data block moving subunit is used for storing the data block into a new storage medium with the medium capability higher than that of the current storage medium when the liability value is greater than the preset threshold value.
In some embodiments of the invention, the apparatus further comprises:
the data block queue to be written generating module is used for acquiring the data values of all the data blocks to be written and arranging all the data blocks from high to low according to the data value corresponding to each data block to generate a data block queue to be written;
the medium queue generating module is used for acquiring the medium capacities of all the storage media and arranging all the storage media from high to low to generate a medium queue according to the medium capacity corresponding to each storage medium;
the data block writing module is used for writing the data block with the highest data value in the data block queue into the storage medium with the highest capacity in the medium queue;
and the medium queue updating module is used for deleting the written data blocks from the data block queue to be written, and transferring to the step of acquiring the medium capacities of all the storage media and arranging all the storage media from high to low according to the medium capacity corresponding to each storage medium to generate a medium queue until no data block to be written exists in the data block queue to be written.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory for storing one or more programs; a processor. The program or programs, when executed by a processor, implement the method of any of the first aspects as described above.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the method according to any one of the first aspect described above.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a flowchart of a distributed storage method according to an embodiment of the present invention;
fig. 2 is a block diagram of a distributed storage apparatus according to an embodiment of the present invention;
fig. 3 is a block diagram of an electronic device according to an embodiment of the present invention.
Icon: 100-distributed storage; 110-a data acquisition module; 120-alignment module; 130-a storage management module; 101-a memory; 102-a processor; 103-communication interface.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the individual features of the embodiments can be combined with one another without conflict.
In the storage medium flow type storage mode in the prior art, a medium with a higher level is bound to a medium with a lower level by using software or an algorithm to form a mixed medium, data is stored in the virtual mixed medium, and flow and exchange are performed in two different physical media by using a fixed algorithm. This method can more efficiently perform data flow and exchange, but has a serious defect in the management of various storage media, such as that a certain medium cannot be freely and independently expanded or a new medium type is expanded, and the expansion process still needs to expand a mixed medium in a basic proportion. Storage pool streaming storage offers certain advances over the previous storage pool streaming in terms of media replacement and expansion, by streaming and swapping data between different storage pools, which typically use different storage media. The mainstream method is to flow data between two storage pools, and some products are designed into a plurality of storage pools. The storage pool streaming type mode has a common defect in terms of data streaming and exchange timing, for example, ceph has a problem of amplified reading when transferring data from a low-level storage pool to a high-level storage pool, and distributed storage needs to read data of a minimum logical block from the low-level storage pool to the high-level storage pool no matter how small the actually required data block is. Storage copy streaming storage refers to allowing multiple copies of distributed storage to be deposited on different media. For example, the distributed storage adopts a three-copy protection mode, the system stores the first two copies into a high-level medium, stores the last copy into a lower-level medium, and returns the first two copies to the client after the writing is completed. The method solves the problem of fixing the medium model to a great extent, basically can realize the expansion of the medium without correlation, but has larger performance influence when the medium with lower performance can not bear under the condition of slightly higher performance requirement; in addition, the safety of data is sacrificed to a certain extent; most critically, this approach does not yield high gains in available capacity.
Therefore, a distributed storage method is provided, which has no requirement for a fixed mode for disk media, and different disk media can be automatically arranged for storage for different data systems, and data can be stored in the disk media in an accurate and efficient flowing manner.
Referring to fig. 1, fig. 1 is a flowchart of a distributed storage method according to an embodiment of the present invention, where the distributed storage method includes the following steps:
step S110: and acquiring the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium.
The medium capability of the storage medium may include at least one of a performance parameter, a lifetime, a storage status, a space size, etc. of the storage medium. The data value of the data block stored in the storage medium can be represented according to at least one of the utilization rate of the data block, the importance degree of the data and the like.
Step S120: comparing the medium capacity with the data value to obtain a comparison result.
After the media capability of the storage medium at the current time and the data value of the data block stored in the storage medium are obtained, the media capability and the data value can be compared to obtain a comparison result indicating whether the media capability of the storage medium is matched with the data value of the data block.
Step S130: and carrying out storage management on the data blocks according to the comparison result.
And finally, carrying out storage management on the data blocks according to the comparison result so as to realize the storage management of matching the data value of the data blocks with the medium capacity of the storage medium, thereby ensuring that the storage medium has enough capacity to support the data blocks stored therein, and further ensuring that the storage medium is effectively managed so as to enable the storage medium to exert the maximum storage performance.
In the implementation process, the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium are firstly obtained, and the medium capability and the data value are compared to judge whether the benefit maximization is realized when the data block is stored in the storage medium at the current moment. And then, carrying out storage management on the data block according to the comparison result, and if the medium capacity is not matched with the data value, allocating a proper storage medium for the data block again for storage, thereby realizing accurate and efficient flow storage.
In some embodiments of the present invention, when the medium capacity of the storage medium at the current time is obtained, the attribute parameter, the health degree, the capacity space, and the busy degree of the storage medium may be obtained first, and then the attribute parameter, the health degree, the capacity space, and the busy degree are weighted respectively to calculate the medium capacity of the storage medium at the current time.
The media capabilities may depend on the storage media's intrinsic property capabilities (Attributes), health (health), capacity space (capacity), busyness (busy) four data. Where the inherent property capabilities of the storage medium depend on the disk type and interface. For example, the disk types mainly include a mechanical hard disk (HDD), a solid state hard disk (SSD), and a hybrid hard disk (SSHD), the interfaces mainly include IDE, SATA, SCSI, SAS, m.2, U.2, PCIE, HHD disk configured with IDE, SATA, SCSI, SAS, m.2, U.2, PCIE interfaces with inherent attribute capabilities of 1, 2, 3, 4, 5, 6, 7, SSD disk configured with IDE, SATA, SCSI, SAS, m.2, U.2, PCIE interfaces with inherent attribute capabilities of 12, 13, 14, 15, 16, 17, 18, SSD disk configured with IDE, SATA, SCSI, SAS, m.2, U.2, PCIE interfaces with inherent attribute capabilities of 12, 13, 14, 15, 16, 17, 18, SSHD disk configured with IDE, SATA, SCSI, SAS, m.2, U.2, PCIE with inherent attribute capabilities of 5, 6, 7, 8, 9, 10, 11, respectively. The health level (health), capacity space (capacity), and busy level (busy) can be obtained by capturing the real-time status of the storage medium. And then, integrating four factors of inherent attribute capability, health degree, capacity space and busy degree of the medium, and performing weighted calculation to obtain the medium capability available at the current moment of the disk. It will be appreciated that the above described solution is only one possible implementation and that different types of parameters relating to the medium capabilities of the storage medium may be chosen according to specific requirements.
In some embodiments of the present invention, when the data value of the data block stored in the storage medium is obtained, an I/O model value influence value, a service importance value influence value, and a read-write hit rate value influence value of the data block may be obtained first; and then calculating the data value of the data block according to the data block value quantity calculation model, the I/O model value influence value of the data block, the service importance value influence value and the read-write hit rate value influence value.
Specifically, the value influence value of the I/O model may be obtained according to the following first relation function:
then, constant parameters specified when allocating memory space are acquired
As a business importance value impact value; wherein the content of the first and second substances,
；
then, obtaining a value influence value of the read-write hit rate according to the following second relation function:
finally, calculating the value influence value of the model and the I/O model of the data block according to the value quantity of the data block
Business importance value impact value
And read-write hit rate value impact value
Calculating the data value of the data block:
Wherein, the I/O model mainly represents the block size of the I/O, random I/O or sequential I/O from different latitudes. Small block I/O is generally of greater importance than large block I/O in a production environment, random I/O is of greater importance than sequential I/O, and when I/O is large enough, the performance of lower ranked disks will be closer and closer in both random and sequential terms. The I/O model cost impact value may thus be calculated by a first relationship function, e.g.,
which represents the size of the data block or blocks,
indicating the data block read type, if the data block is random I/O
If the data block is sequential I/O, then
. As can be appreciated, the first and second electrodes,
the value can also be set and obtained according to other modes.
The higher the frequency of reusing the data block is, the higher the value of the data block is, and generally, the highest value of the data block is when the data is written and stored for the first time, and as time goes by, the frequency of using the data block is reduced relative to the frequency of using the data block when the data is written for the first time, and therefore, the value is also reduced, so that the relationship function between the value of the data block and the writing time and the frequency of using the data block can be defined by the second relationship function.
After the parameters related to the value of the data block are obtained, a data block value quantity calculation model is comprehensively established to calculate the data value. In the data block worth amount calculation model,
representing the weight occupied by the value impact value of the I/O model,
representing the weight occupied by the business importance value impact value,
the weight occupied is represented, and the value influence value of the read-write hit rate is represented. The value influence of the three indexes on the data block is considered to be equal, namely the three indexes are obtained
The weights corresponding to the indexes may also be defined according to actual requirements, and are not specifically limited herein.
After the comparison result is obtained, the storage management of the data block according to the comparison result may include the following processes. If the comparison result shows that the data value is reduced, the data block is stored in a new storage medium with the medium capability lower than that of the current storage medium, and if the comparison result shows that the data value is reduced, the data value is lower than the medium capability, the medium capability of the storage medium is not matched with the data value of the data block, and the data block is stored in the storage medium, which causes the waste of the performance of the storage medium, so that the data block can be stored in the storage medium with the lower value capability, namely, the data block is stored in the storage medium matched with the current data value. If the comparison result shows that the data value is increased, namely the data block in the storage medium obtains more reading and writing frequency, the heat degree is increased or the data value is increased, according to the logic of maximizing the benefit of the storage medium, the storage medium does not give up the data block actively, the storage medium can grasp the data block to enlarge the benefit of the storage medium, the liability condition of the current storage medium is recorded, and the storage management is carried out on the data block according to the liability condition.
In some embodiments of the present invention, the step of recording the liability condition of the current storage medium, and performing storage management on the data block according to the liability condition comprises: when the response of the current storage medium is not timely in the process of operating the data block, increasing the liability value of the current storage medium; and when the liability value is larger than a preset threshold value, storing the data block into a new storage medium with the medium capability higher than that of the current storage medium.
If the data blocks in the storage medium are depreciated, the data blocks can be flowed to a lower-grade storage medium, but if the data blocks in the storage medium obtain more reading and writing frequencies, the heat degree is high, the data value is high, according to the logic of maximizing the benefit of the storage medium, the storage medium still catches the data blocks to enlarge the benefit of the storage medium, but if the data blocks do not respond timely in the value-added process, the storage medium is charged, and in order to eliminate the charge, the storage medium needs to transfer the high-value data blocks out of other high-grade storage media to eliminate the charge of the storage medium, so that the situation that the storage medium cannot bear the reading and writing operations of the data blocks with high data values is effectively avoided, and the high efficiency of data storage is further ensured.
When a data market receives a client data writing request, a data block distribution writing process is required, the data values of all data blocks to be written can be obtained firstly, and all data blocks are arranged from high to low according to the data value corresponding to each data block to generate a data block queue to be written; then, acquiring the medium capacity of all the storage media, and arranging all the storage media from high to low according to the medium capacity corresponding to each storage medium to generate a medium queue; writing the data block with the highest data value in the data block queue into the storage medium with the highest capacity in the medium queue; and deleting the written data blocks from the data block queue to be written, and transferring to the step of acquiring the medium capacity of all the storage media and arranging all the storage media from high to low according to the medium capacity corresponding to each storage medium to generate a medium queue until no data block to be written exists in the data block queue to be written.
In addition, when the data block is subjected to storage management according to the comparison result, the data block of which the data value is not matched with the medium capability of the storage medium may be sent to the data market according to the comparison result, and then the data block may be subjected to the storage management operation of rewriting through the above process. By the scheme, the matching condition of the medium capacity and the data value of the data block in the storage medium can be monitored regularly, and the unmatched data block is screened and released to the data market again; through data value calculation and medium capacity evaluation, data market recirculation or newly received data blocks are written into the most appropriate storage medium in real time, and therefore benefit maximization between data and the storage medium is guaranteed.
By the method, a data block value calculation model is comprehensively constructed according to three indexes of an I/O model value influence value, a service importance value influence value and a read-write hit rate value influence value of a data block, the medium capacity is weighted and evaluated according to four factors of inherent attribute capacity, health degree, capacity space and busy degree of a medium, the value of the stored data block on the medium is calculated by using the data block value calculation model, the data block which is not matched with the medium capacity is screened and redistributed to a data market for circulation, a new data block write-in request received by the data market and the data block redistributed to the data market are written into a storage medium with the highest medium capacity after real-time updating and evaluation in a system in sequence from high to low according to the principle of medium benefit maximization; through the value circulation type distributed storage mode, the data block can automatically and preferentially select the storage medium with the best performance for storage, so that any storage medium can be effectively managed, the storage performance of the storage medium can be maximized, the maximization of medium benefits is realized, meanwhile, a client can be liberated from the previous configuration and maintenance work, and the intellectualization is realized.
Based on the same inventive concept, the present invention further provides a distributed storage apparatus 100, please refer to fig. 2, and fig. 2 is a block diagram of a distributed storage apparatus 100 according to an embodiment of the present invention.
The distributed storage apparatus 100 includes:
a data obtaining module 110, configured to obtain a media capability of the storage medium at the current time and a data value of a data block stored in the storage medium;
a comparison module 120 for comparing the medium capability and the data value to obtain a comparison result;
and the storage management module 130 is configured to perform storage management on the data block according to the comparison result.
In some embodiments of the present invention, the data acquisition module 110 comprises:
a storage medium parameter acquiring unit for acquiring attribute parameters, health degree, capacity space and busy degree of the storage medium;
and the medium capacity calculating unit is used for weighting the attribute parameters, the health degree, the capacity space and the busy degree respectively so as to calculate and obtain the medium capacity of the storage medium at the current moment.
In some embodiments of the present invention, the data acquisition module 110 comprises:
the data block parameter acquisition unit is used for acquiring an I/O model value influence value, a service importance value influence value and a read-write hit rate value influence value of the data block;
and the data value calculation unit is used for calculating the data value of the data block according to the data block value quantity calculation model, the I/O model value influence value of the data block, the service importance value influence value and the read-write hit rate value influence value.
In some embodiments of the present invention, the data acquisition module 110 comprises:
an I/O model value influence value acquisition unit for acquiring an I/O model value influence value according to a first relationship function:
a business importance value influence value acquisition unit for acquiring constant parameters specified when allocating storage space
As a business importance value impact value; wherein the content of the first and second substances,
；
a reading and writing hit rate value influence value obtaining unit, configured to obtain a reading and writing hit rate value influence value according to the following second relation function:
a data value acquisition unit for calculating the value influence value of the I/O model and the service importance value influence value of the model and the data block according to the value of the data block
And calculating the data value of the data block according to the read-write hit rate value influence value:
In some embodiments of the present invention, storage management module 130 comprises:
the first storage management unit is used for storing the data blocks into a new storage medium with the medium capability lower than that of the current storage medium if the comparison result shows that the data value is reduced;
and the second storage management unit is used for recording the liability condition of the current storage medium and carrying out storage management on the data block according to the liability condition if the comparison result shows that the data value is increased.
In some embodiments of the invention, the second storage management unit comprises:
the liability value increasing subunit is used for increasing the liability value of the current storage medium when the response of the current storage medium is not timely in the process of operating the data block;
and the data block moving subunit is used for storing the data block into a new storage medium with the medium capability higher than that of the current storage medium when the liability value is greater than the preset threshold value.
In some embodiments of the invention, the apparatus further comprises:
the data block queue to be written generating module is used for acquiring the data values of all the data blocks to be written and arranging all the data blocks from high to low according to the data value corresponding to each data block to generate a data block queue to be written;
the medium queue generating module is used for acquiring the medium capacities of all the storage media and arranging all the storage media from high to low to generate a medium queue according to the medium capacity corresponding to each storage medium;
the data block writing module is used for writing the data block with the highest data value in the data block queue into the storage medium with the highest capacity in the medium queue;
and the medium queue updating module is used for deleting the written data blocks from the data block queue to be written, and transferring to the step of acquiring the medium capacities of all the storage media and arranging all the storage media from high to low according to the medium capacity corresponding to each storage medium to generate a medium queue until no data block to be written exists in the data block queue to be written.
Referring to fig. 3, fig. 3 is a schematic structural block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device comprises a memory 101, a processor 102 and a communication interface 103, wherein the memory 101, the processor 102 and the communication interface 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The memory 101 may be used for storing software programs and modules, such as program instructions/modules corresponding to the distributed storage apparatus 100 provided in the embodiments of the present application, and the processor 102 executes the software programs and modules stored in the memory 101, so as to execute various functional applications and data processing. The communication interface 103 may be used for communicating signaling or data with other node devices.
The Memory 101 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
The processor 102 may be an integrated circuit chip having signal processing capabilities. The Processor 102 may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
It will be appreciated that the configuration shown in fig. 3 is merely illustrative and that the electronic device may include more or fewer components than shown in fig. 3 or have a different configuration than shown in fig. 3. The components shown in fig. 3 may be implemented in hardware, software, or a combination thereof.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In summary, the distributed storage method and apparatus provided in the embodiments of the present application, the method includes: acquiring the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium; comparing the medium capacity with the data value to obtain a comparison result; and carrying out storage management on the data blocks according to the comparison result. The method comprises the steps of firstly obtaining the medium capacity of the storage medium at the current moment and the data value of a data block stored in the storage medium, and comparing the medium capacity and the data value to judge whether the benefit maximization is realized when the data block is stored in the storage medium at the current moment. And then, carrying out storage management on the data block according to a comparison result, if the medium capacity is not matched with the data value, allocating a proper storage medium for the data block again for storage, namely, comparing the medium capacity of the storage medium with the data value of the data block, and finally achieving higher efficiency and accuracy of storage and flow of the data block in distributed storage, so that the data block can go to a place where the data block should go most when necessary, thereby realizing medium benefit maximization and realizing accurate and efficient flow storage.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (10)
1. A distributed storage method, the method comprising:
acquiring the medium capacity of a storage medium at the current moment and the data value of a data block stored in the storage medium;
comparing the medium capacity with the data value to obtain a comparison result;
and carrying out storage management on the data blocks according to the comparison result.
2. The method of claim 1, wherein the step of obtaining the media capabilities of the storage medium at the current time comprises:
acquiring attribute parameters, health degree, capacity space and busyness degree of the storage medium;
and weighting the attribute parameters, the health degree, the capacity space and the busy degree respectively to calculate and obtain the medium capacity of the storage medium at the current moment.
3. The method of claim 1, wherein the step of obtaining a data value of the data block stored in the storage medium comprises:
obtaining an I/O model value influence value, a service importance value influence value and a read-write hit rate value influence value of the data block;
and calculating the data value of the data block according to a data block value calculation model, the I/O model value influence value of the data block, the business importance value influence value and the read-write hit rate value influence value.
4. The method of claim 1, wherein the step of obtaining a data value of the data block stored in the storage medium comprises:
obtaining an I/O model value influence value according to the following first relation function:
obtaining constant parameters specified when allocating storage space
As a business importance value impact value; wherein the content of the first and second substances,
；
and obtaining a value influence value of the read-write hit rate according to the following second relation function:
calculating a model according to the value of the data block, and calculating the value influence value of the I/O model and the service importance value influence value of the data block
And read-write hit rate value impact value
Counting data blocksAccording to the value:
5. The method of claim 1, wherein the step of performing storage management on the data block according to the comparison result comprises:
if the comparison result shows that the data value is reduced, storing the data block into a new storage medium with the medium capability lower than that of the current storage medium;
and if the comparison result shows that the data value is increased, recording the liability condition of the current storage medium, and performing storage management on the data block according to the liability condition.
6. The method of claim 5, wherein recording the liability condition of the current storage medium, and wherein the step of performing storage management on the data block according to the liability condition comprises:
when the current storage medium does not respond in time in the process of operating the data block, increasing the liability value of the current storage medium;
and when the liability value is larger than a preset threshold value, storing the data block into a new storage medium with the medium capability higher than that of the current storage medium.
7. The method of claim 1, wherein the method further comprises:
acquiring data values of all data blocks to be written, and arranging all the data blocks from high to low according to the data value corresponding to each data block to generate a data block queue to be written;
acquiring the medium capacity of all storage media, and arranging all the storage media from high to low according to the medium capacity corresponding to each storage medium to generate a medium queue;
writing the data block with the highest data value in the data block queue into a storage medium with the highest capacity in a medium queue;
deleting the written data blocks from the data block queue to be written, and transferring to the step of acquiring the medium capacities of all the storage media and arranging all the storage media from high to low according to the medium capacity corresponding to each storage medium to generate a medium queue until no data block to be written exists in the data block queue to be written.
8. A distributed storage apparatus, the apparatus comprising:
the data acquisition module is used for acquiring the medium capability of the storage medium at the current moment and the data value of the data block stored in the storage medium;
the comparison module is used for comparing the medium capacity and the data value to obtain a comparison result;
and the storage management module is used for carrying out storage management on the data block according to the comparison result.
9. An electronic device, comprising:
a memory for storing one or more programs;
a processor;
the one or more programs, when executed by the processor, implement the method of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110158659.XA CN112486423A (en) | 2021-02-05 | 2021-02-05 | Distributed storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110158659.XA CN112486423A (en) | 2021-02-05 | 2021-02-05 | Distributed storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112486423A true CN112486423A (en) | 2021-03-12 |
Family
ID=74912372
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110158659.XA Pending CN112486423A (en) | 2021-02-05 | 2021-02-05 | Distributed storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112486423A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114281260A (en) * | 2021-12-23 | 2022-04-05 | 广州炒米信息科技有限公司 | Storage method, device, equipment and medium applied to distributed storage system |
| CN114691698A (en) * | 2022-04-24 | 2022-07-01 | 北京梦蓝杉科技有限公司 | Data processing system and method for computer system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605615A (en) * | 2013-11-21 | 2014-02-26 | 浪潮电子信息产业股份有限公司 | Block-level-data-based directional allocation method for hierarchical storage |
| CN105205014A (en) * | 2015-09-28 | 2015-12-30 | 北京百度网讯科技有限公司 | Data storage method and device |
| CN110737391A (en) * | 2018-07-20 | 2020-01-31 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for managing a storage system |
| CN111124269A (en) * | 2018-10-31 | 2020-05-08 | 伊姆西Ip控股有限责任公司 | Method, electronic device, and computer-readable storage medium for storage management |
-
2021
- 2021-02-05 CN CN202110158659.XA patent/CN112486423A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605615A (en) * | 2013-11-21 | 2014-02-26 | 浪潮电子信息产业股份有限公司 | Block-level-data-based directional allocation method for hierarchical storage |
| CN105205014A (en) * | 2015-09-28 | 2015-12-30 | 北京百度网讯科技有限公司 | Data storage method and device |
| CN110737391A (en) * | 2018-07-20 | 2020-01-31 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for managing a storage system |
| CN111124269A (en) * | 2018-10-31 | 2020-05-08 | 伊姆西Ip控股有限责任公司 | Method, electronic device, and computer-readable storage medium for storage management |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114281260A (en) * | 2021-12-23 | 2022-04-05 | 广州炒米信息科技有限公司 | Storage method, device, equipment and medium applied to distributed storage system |
| CN114691698A (en) * | 2022-04-24 | 2022-07-01 | 北京梦蓝杉科技有限公司 | Data processing system and method for computer system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200142587A1 (en) | Automatic tiering of storage using dynamic grouping | |
| JP4837445B2 (en) | Storage system and management apparatus and method | |
| US7478177B2 (en) | System and method for automatic reassignment of shared storage on blade replacement | |
| CN104462389B (en) | Distributed file system implementation method based on classification storage | |
| US8756198B2 (en) | Enhancing data store backup times | |
| JP2007323560A (en) | Performance monitoring method, computer, and computer system | |
| CN112486423A (en) | Distributed storage method and device | |
| US11914894B2 (en) | Using scheduling tags in host compute commands to manage host compute task execution by a storage device in a storage system | |
| US11836067B2 (en) | Hyper-converged infrastructure (HCI) log system | |
| JP6269140B2 (en) | Access control program, access control method, and access control apparatus | |
| JP2010211743A5 (en) | ||
| WO2019084917A1 (en) | Method and apparatus for calculating available capacity of storage system | |
| US20150347253A1 (en) | Federating and Protecting Multiple Big Data Sources Combined with Enterprise Data Sources for BI | |
| CN110019017B (en) | High-energy physical file storage method based on access characteristics | |
| Chang et al. | Dynamic deduplication decision in a hadoop distributed file system | |
| CN108845771A (en) | Method, system and the associated component of RAID are configured in a kind of storage system | |
| US20220391370A1 (en) | Evolution of communities derived from access patterns | |
| US11200210B2 (en) | Method of efficient backup of distributed file system files with transparent data access | |
| US20230137673A1 (en) | Systems and methods for dynamically scaling remote resources | |
| US9690886B1 (en) | System and method for a simulation of a block storage system on an object storage system | |
| US11580082B2 (en) | Object storage system with control entity quota usage mapping | |
| US11836378B2 (en) | Storage system | |
| Luo et al. | A new big data storage architecture with intrinsic search engines | |
| US10042572B1 (en) | Optimal data storage configuration | |
| US11983431B2 (en) | Read-disturb-based read temperature time-based attenuation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210312 |
|
| RJ01 | Rejection of invention patent application after publication |