CN116737380B - Balanced storage method, device and equipment for distributed memory and storage medium - Google Patents
Balanced storage method, device and equipment for distributed memory and storage medium Download PDFInfo
- Publication number
- CN116737380B CN116737380B CN202310717570.1A CN202310717570A CN116737380B CN 116737380 B CN116737380 B CN 116737380B CN 202310717570 A CN202310717570 A CN 202310717570A CN 116737380 B CN116737380 B CN 116737380B
- Authority
- CN
- China
- Prior art keywords
- storage
- result
- stored
- data
- balanced
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 29
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 8
- 238000005457 optimization Methods 0.000 abstract description 4
- 238000013500 data storage Methods 0.000 abstract description 2
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed memory balanced storage method, a device, equipment and a storage medium, and belongs to the technical field of data storage. Firstly, acquiring data to be stored; reading the storage state of the distributed memory and outputting a storage state result; performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result; and finally, storing the data to be stored according to the balanced storage result. According to the storage state of the existing distributed storage, the method uses the equilibrium storage algorithm to carry out equilibrium storage optimization on the file storage process of the distributed storage, changes the file storage mode of the distributed storage, and greatly improves the equilibrium stability of the utilization rate of the distributed storage disk.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a method, an apparatus, a device, and a storage medium for balanced storage of a distributed memory.
Background
The distributed storage is a technology for jointly providing storage service by a plurality of nodes through networking connection, and when data is stored specifically, the distributed storage can be used for blocking and backing up the data and then dispersing the data to different servers. However, in reality, the disk sizes of each physical node of the server may not be the same, so that when the data is stored in blocks, the data distribution on the disks is easily unbalanced, some disks are nearly full, and some other disks may be more idle.
In the existing scheme, a problem of balanced distributed storage and storage is generally solved by adopting a mechanism of ceph in an open source product, the mechanism of ceph is firstly used for giving weight to the capacity of a disk and then carrying out a pseudo-random calculation mode by adding parameters such as a fault domain and the like, and the mode can achieve good effect on balance, but can cause a large amount of unbalance among the disks under extreme conditions. Therefore, how to make full use of the magnetic disk and make the usage rate of all magnetic disks more stable and balanced becomes a technical problem to be solved.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide a method, a device, equipment and a storage medium for balanced storage of a distributed memory, and aims to solve the technical problem of unbalanced disk storage utilization rate during distributed storage in the prior art.
In order to achieve the above object, the present invention provides a distributed memory balanced storage method, which includes the following steps:
acquiring data to be stored;
reading the storage state of the distributed memory and outputting a storage state result;
performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result;
and storing the data to be stored according to the balanced storage result.
Optionally, reading the storage state of the distributed memory and outputting the storage state result, and the specific steps include:
reading the server state of the distributed memory and outputting an available server result;
carrying out available hard disk statistics according to the available server result and outputting a statistics result;
and obtaining and outputting the storage state result according to the available server result and the statistical result.
Optionally, performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result, and specifically, the steps include:
acquiring storage state data according to the storage state result;
and inputting the storage state data and the data to be stored into an equalization storage algorithm to obtain and output the equalization storage result.
Optionally, inputting the storage state data and the data to be stored into an equalization storage algorithm, and obtaining and outputting the equalization storage algorithm in the equalization storage result, wherein the specific steps include:
analyzing the data to be stored to obtain a data backup result to be stored;
and calculating the number of PG to be stored of the unit hard disk according to the backup result of the data to be stored and the storage state data, and outputting a calculation result, wherein the number of PG to be stored of the unit hard disk is as follows:
PGs＝n×k/m
the PGs are the number of PGs to be stored in a unit hard disk, n is the total storage space division number, k is the number of data backup, and m is the total number of hard disks.
Optionally, after calculating the number of PGs to be stored in the unit hard disk according to the backup result of the data to be stored and the storage status data and outputting the calculation result, the method further includes:
and performing linear programming operation of the balanced storage result according to the calculation result to obtain and output a linear programming operation result, wherein the linear programming is as follows:
s.t.：PG i ≤PGs
PG i ≥0
wherein Max is the maximum value of the sum of PG numbers of all hard disks, i is the number of the hard disks, m is the total number of the hard disks, PG i The number of PGs in the ith hard disk is the number of PGs to be stored in the unit hard disk.
Optionally, inputting the storage state data and the data to be stored into an equalization storage algorithm, obtaining and outputting the equalization storage algorithm in the equalization storage result, and the specific steps further include:
receiving the linear programming operation result and analyzing and judging;
when the linear programming operation result is a non-integer, rounding the linear programming operation result to obtain the balanced storage result;
and when the linear programming operation result is an integer, taking the linear programming operation result as the balanced storage result.
Optionally, storing the data to be stored according to the balanced storage result specifically includes:
analyzing the balanced storage result to obtain a balanced storage strategy;
and storing the data to be stored according to the balanced storage strategy.
In addition, in order to achieve the above object, the present invention also proposes a distributed memory balanced storage device, including:
and a data acquisition module: acquiring data to be stored;
and a state reading module: reading the storage state of the distributed memory and outputting a storage state result;
and the equalization algorithm module is used for: performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result;
and the storage execution module is used for: and storing the data to be stored according to the balanced storage result.
In addition, to achieve the above object, the present invention also proposes a distributed memory balancing storage device, including: a memory, a processor, and a distributed memory balancing storage program stored on the memory and executable on the processor, the distributed memory balancing storage program configured to implement the steps of the distributed memory balancing storage method as described above.
In addition, to achieve the above object, the present invention also proposes a computer-readable storage medium storing a computer program, the storage medium storing thereon a distributed memory balancing storage program which, when executed by a processor, implements the steps of the distributed memory balancing storage method as described above.
The invention obtains the data to be stored; reading the storage state of the distributed memory and outputting a storage state result; performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result; and finally, storing the data to be stored according to the balanced storage result. According to the storage state of the existing distributed storage, the method uses the equilibrium storage algorithm to carry out equilibrium storage optimization on the file storage process of the distributed storage, changes the file storage mode of the distributed storage, and greatly improves the equilibrium stability of the utilization rate of the distributed storage disk.
Drawings
FIG. 1 is a schematic diagram of a distributed memory balancing storage device of a hardware runtime environment according to an embodiment of the present invention;
FIG. 2 is a flow chart of a first embodiment of a method for balancing memory of the present invention;
FIG. 3 is a flow chart illustrating a second embodiment of a method for balancing storage of a distributed memory according to the present invention;
FIG. 4 is a flow chart illustrating a third embodiment of a balanced storage method of the distributed memory according to the present invention;
FIG. 5 is a flow chart of a fourth embodiment of a balanced storage method of the distributed memory according to the present invention;
FIG. 6 is a block diagram of a first embodiment of a distributed memory balanced storage device according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a distributed memory balancing storage device of a hardware running environment according to an embodiment of the present invention.
As shown in fig. 1, the distributed memory balancing storage device may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Those skilled in the art will appreciate that the architecture shown in fig. 1 is not limiting of a distributed memory balanced storage device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a distributed memory balance storage program may be included in the memory 1005 as one type of storage medium.
In the distributed memory balancing storage device shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the distributed memory balancing storage device of the present invention may be disposed in the distributed memory balancing storage device, where the distributed memory balancing storage device calls a distributed memory balancing storage program stored in the memory 1005 through the processor 1001, and executes the distributed memory balancing storage method provided by the embodiment of the present invention.
An embodiment of the present invention provides a method for balanced storage of a distributed memory, and referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of a method for balanced storage of a distributed memory according to the present invention.
In this embodiment, the method for balanced storage of the distributed memory includes the following steps:
step S10: acquiring data to be stored;
it should be noted that, when the data to be stored is obtained after the files to be stored are parsed and backed up in blocks, in a specific implementation, the files to be stored are parsed to obtain the file type, the file size and other storage content information of the files to be stored, and then the files to be stored are backed up in blocks to obtain corresponding data blocks of the files to be stored, and in general, the data blocks of the files to be stored are respectively stored through different server hard disks in a distributed manner, so that the data backup storage of the contents to be stored is finally realized.
Step S20: reading the storage state of the distributed memory and outputting a storage state result;
it is understood that the reading of the storage state of the distributed storage refers to the reading statistics of the number of storable servers of the existing distributed storage and the number of storable hard disks in the storable servers, so as to realize the knowledge of the overall storage capacity status of the distributed storage.
Step S30: performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result;
in particular, the balanced storage processing refers to balanced distribution storage of data to be stored based on the existing state of the distributed storage, so as to ensure that the storage state of each storable hard disk of the distributed storage is balanced and stable after the data to be stored is stored in the distributed storage.
Step S40: and storing the data to be stored according to the balanced storage result.
It will be appreciated that the balanced storage result is a storage policy for performing balanced distributed storage on the data to be stored, that is, the distributed storage will store the data to be stored according to the storage policy.
The embodiment obtains the data to be stored; reading the storage state of the distributed memory and outputting a storage state result; performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result; and finally, storing the data to be stored according to the balanced storage result. According to the storage state of the existing distributed storage, the file storage process of the distributed storage is subjected to balanced storage optimization by using an balanced storage algorithm, so that the file storage mode of the distributed storage is changed, and the balanced stability of the utilization rate of a distributed storage disk is greatly improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating a second embodiment of a method for balancing storage of a distributed memory according to the present invention.
Based on the first embodiment, in this embodiment, the step S20 further includes:
step S21: reading the server state of the distributed memory and outputting an available server result;
it will be appreciated that the available server results refer to servers that have a relative amount of storage space remaining in the existing overall distributed storage and have a relative capacity, where a server that has a relative amount of storage space remaining refers specifically to a server that has a lower utilization of the existing storage space than other storage servers.
Step S22: carrying out available hard disk statistics according to available server results and outputting statistics results;
it should be noted that, generally, a server includes a plurality of hard disks, but specific storage states of the hard disks in the same server are not the same, so in order to ensure an equilibrium storage state of the overall distributed memory, storage space utilization of each storage hard disk needs to be maintained approximately equal, so that the available hard disk specifically refers to a hard disk that has a relative storage space remaining in the existing overall distributed memory and has a relative storage capacity, where a hard disk that has a relative storage space remaining specifically refers to a storage hard disk that has a lower utilization of the existing storage space than other storage hard disks.
Step S23: and obtaining and outputting a storage state result according to the available server result and the statistical result.
It will be appreciated that storing the state results specifically includes: before the current storage operation is executed, the number of available servers and specific server numbers, and the number of existing storage hard disks and specific storage hard disk numbers are used.
According to the embodiment, the storage space state of the existing distributed storage, which can execute the storage operation, is obtained by reading the storage states of each server and the corresponding storage hard disk in the existing distributed storage, so that the effective utilization rate of the storage space of the distributed storage is improved.
Referring to fig. 4, fig. 4 is a flowchart illustrating a third embodiment of a balanced storage method of a distributed memory according to the present invention.
Based on the first embodiment, in this embodiment, the specific steps of step S30 include:
step S31: acquiring storage state data according to the storage state result;
it should be noted that, the storage status data specifically includes the total number m of available storage hard disks in the balanced storage algorithm.
Step S32: and inputting the storage state data and the data to be stored into an equalization storage algorithm to obtain and output an equalization storage result.
It should be noted that, the data to be stored specifically includes the data backup number k in the balanced storage algorithm.
According to the embodiment, through reading the storage state, the input data of the balanced storage algorithm are obtained, and starting conditions are provided for the subsequent execution of the balanced storage algorithm.
Referring to fig. 5, fig. 5 is a flowchart illustrating a fourth embodiment of a balanced storage method of a distributed memory according to the present invention.
Based on the above third embodiment, in this embodiment, the specific steps of step S32 include:
step S321: analyzing the data to be stored to obtain a data backup result to be stored;
it will be appreciated that the data backup result to be stored specifically refers to the number of samples of the same stored data that need to be subjected to a persistent backup in the entire distributed storage.
Step S322: according to the backup result of the data to be stored and the storage state data, calculating the number of PG to be stored of the unit hard disk, and outputting the calculation result, wherein the number of PG to be stored of the unit hard disk is as follows:
PGs＝n×k/m
the PGs are the number of PGs to be stored in a unit hard disk, n is the total storage space division number, k is the number of data backup, and m is the total number of hard disks.
Note that, the total storage space division number n is a parameter adaptively set according to the specific size of the overall distributed storage space, if the total storage space is increased, the total storage space division number is correspondingly increased, and if the total storage space is reduced, the total storage space division number is correspondingly reduced. For example, when the total storage space is made up of six standard storage hard disks of three servers, the total storage space division number n may be set to 256.
Step S323: and performing linear programming operation of the balanced storage result according to the calculation result to obtain and output a linear programming operation result, wherein the linear programming is as follows:
s.t.：PG i ≤PGs
PG i ≥0
wherein Max is the maximum value of the sum of PG numbers of all hard disks, i is the number of the hard disks, m is the total number of the hard disks, PG i The number of PGs in the ith hard disk is the number of PGs to be stored in the unit hard disk.
It should be noted that the linear planning operation in the equilibrium storage algorithm is formed based on three constraint conditions, which are respectively: ensuring that the sum of the PG numbers on all hard disks is maximum; ensuring the PG quantity on each disk and the PG quantity which does not exceed the unit hard disk to be stored; the number of storage per hard disk cannot be negative.
According to the embodiment, through a specific balanced storage algorithm and a corresponding linear programming idea, balanced storage design is carried out on data to be stored, a corresponding operation result is finally obtained, and data support is provided for realizing the balanced storage state of the distributed memory.
In one embodiment, assuming there are three servers A, B and C assuming a distributed storage, 2 hard disks on each server, labeled 1-6 respectively, each hard disk can only select other nodes as a combination item in order to guarantee redundancy of data on each disk. For example, the hard disk 1 can only have 4 combinations of (1, 3) (1, 4) (1, 5) (1, 6), and each hard disk has 4 combinations, and the combinations are selected from the combinations for calculation. When the distributed memory is used for storing a file to be stored, assuming that the whole storage space is divided into 256 PGs and then a dual redundancy mode is adopted, pgs=256×2/6= 85.333333 needs to be stored in each hard disk.
Then inputting the data into an equilibrium storage algorithm, wherein the selected hard disk combination and the conditions to be met are as follows:
condition 1, ensuring that the sum of the numbers of PGs on all hard disks is maximum
Max:PG_1_4+PG_2_5+PG_3_2+PG_4_2+PG_5_4+PG_6_1+PG_3_5+PG_4_6+PG_2_3+PG_1_3+PG_2_4+PG_4_5+PG_3_6+PG_1_5+PG_1_6+PG_2_6；
Condition 2: ensuring that the sum of PG numbers on each disk does not exceed 85.3333
PG_1_4+PG_6_1+PG_1_3+PG_1_5+PG_1_6≤85.33333333333333；
PG_2_5+PG_3_2+PG_4_2+PG_2_3+PG_2_4+PG_2_6≤85.3333333333333；
PG_3_2+PG_3_5+PG_2_3+PG_1_3+PG_3_6≤85.33333333333333；
PG_1_4+PG_4_2+PG_5_4+PG_4_6+PG_2_4+PG_4_5≤85.3333333333333；
PG_2_5+PG_5_4+PG_3_5+PG_4_5+PG_1_5≤85.33333333333333；
PG_6_1+PG_4_6+PG_3_6+PG_1_6+PG_2_6≤85.33333333333333；
Condition 3: third condition: the number of storage per hard disk cannot be negative
PG_1_4≥0；PG_2_5≥0；PG_3_2≥0；PG_4_2≥0；PG_5_4≥0；PG_6_1≥0；
PG_3_5≥0；PG_4_6≥0；PG_2_3≥0；PG_1_3≥0；PG_2_4≥0；PG_4_5≥0；PG_3_6≥0；PG_1_5≥0；PG_1_6≥0；PG_2_6≥0；
The result of linear programming operation using these three constraints is:
PG_1_3＝0；PG_1_4＝85.3333；PG_1_5＝0；PG_1_6＝0；PG_2_3＝0；PG_2_4＝0；
PG_2_5＝85.3333；PG_2_6＝0；PG_3_2＝0；PG_3_5＝0；PG_3_6＝85.3333；PG_4_2＝0；PG_4_5＝0；PG_4_6＝0；PG_5_4＝0；PG_6_1＝0；
and finally, rounding the operation result to obtain a final balanced storage strategy, namely, storing the files to be stored into the hard disk combination respectively: in PG_1_4, PG_2_5 and PG_3_6, 85 PGs are required to be stored for each hard disk combination.
According to the embodiment, through a specific balanced storage algorithm and a corresponding linear programming idea, balanced storage design is carried out on data to be stored, a corresponding operation result is finally obtained, and data support is provided for realizing the balanced storage state of the distributed memory.
Further, the storage state data and the data to be stored are input into an equalization storage algorithm, and the equalization storage algorithm in the equalization storage result is obtained and output, and the specific steps further comprise: receiving a linear programming operation result and analyzing and judging; when the linear programming operation result is a non-integer, rounding the linear programming operation result to obtain an equilibrium storage result; and when the linear programming operation result is an integer, taking the linear programming operation result as an equilibrium storage result.
It should be noted that, since the result of the balanced storage is an integer and is determined by the hardware attribute of the entire distributed memory, when the result of the linear programming operation is a non-integer, it is necessary to perform rounding processing.
Further, storing the data to be stored according to the balanced storage result specifically includes: resolving the balanced storage result to obtain a balanced storage strategy; and storing the data to be stored according to the balanced storage strategy.
It should be noted that, the balanced storage policy refers to storing data to be stored in some specific storage hard disks of the distributed storage, that is, the balanced storage policy includes specific storage hard disk location information.
In addition, the embodiment of the invention also provides a computer readable storage medium storing a computer program, wherein the storage medium stores a distributed memory balanced storage program, and the distributed memory balanced storage program realizes the steps of the distributed memory balanced storage method when being executed by a processor.
Because the storage medium adopts all the technical schemes of all the embodiments, at least county has all the beneficial effects brought by the technical schemes of the embodiments, and the description is omitted here.
Referring to fig. 5, fig. 5 is a block diagram illustrating a first embodiment of a distributed memory balanced storage device according to the present invention.
As shown in fig. 5, the distributed memory balancing storage device according to the embodiment of the present invention includes:
the data acquisition module 10: acquiring data to be stored;
status reading module 20: reading the storage state of the distributed memory and outputting a storage state result;
the equalization algorithm module 30: performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result;
storage execution module 40: and storing the data to be stored according to the balanced storage result.
The embodiment obtains the data to be stored; reading the storage state of the distributed memory and outputting a storage state result; performing balanced storage processing on the data to be stored according to the storage state result and outputting a balanced storage result; and finally, storing the data to be stored according to the balanced storage result. According to the storage state of the existing distributed storage, the file storage process of the distributed storage is subjected to balanced storage optimization by using an balanced storage algorithm, so that the file storage mode of the distributed storage is changed, and the balanced stability of the utilization rate of a distributed storage disk is greatly improved.
It should be understood that the foregoing is illustrative only and is not limiting, and that in specific applications, those skilled in the art may set the invention as desired, and the invention is not limited thereto.
It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the purpose of the embodiment, which is not limited herein.
In addition, technical details not described in detail in this embodiment may refer to the method for balancing storage of the distributed memory provided in any embodiment of the present invention, which is not described herein.
Furthermore, it should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory)/RAM, magnetic disk, optical disk) and including several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (8)
1. A method for balanced storage of a distributed memory, comprising:
acquiring data to be stored;
reading the storage state of the distributed memory and outputting a storage state result;
acquiring storage state data according to the storage state result;
inputting the storage state data and the data to be stored into an equalization algorithm to obtain and output an equalization storage result;
the equalization algorithm specifically comprises the steps of analyzing the data to be stored to obtain a data backup result to be stored;
and calculating the number of PG to be stored of the unit hard disk according to the backup result of the data to be stored and the storage state data, and outputting a calculation result, wherein the number of PG to be stored of the unit hard disk is as follows:
PGs＝n×k/m
wherein PGs are the number of PG to be stored in a unit hard disk, n is the total storage space division number, k is the number of spare parts, and m is the total number of hard disks;
and storing the data to be stored according to the balanced storage result.
2. The method for balanced storage of distributed memory according to claim 1, wherein the steps of reading the storage state of the distributed memory and outputting the result of the storage state include:
reading the server state of the distributed memory and outputting an available server result;
carrying out available hard disk statistics according to the available server result and outputting a statistics result;
and obtaining and outputting the storage state result according to the available server result and the statistical result.
3. The method for balanced storage of distributed storage according to claim 1, wherein after calculating the number of PGs to be stored per hard disk based on the backup result of the data to be stored and the storage status data and outputting the calculation result, further comprising:
and performing linear programming operation of the balanced storage result according to the calculation result to obtain and output a linear programming operation result, wherein the linear programming is as follows:
s.t.：PG i ≤PGs
PG i ≥0
wherein Max is the maximum value of the sum of PG numbers of all hard disks, i is the number of the hard disks, m is the total number of the hard disks, PG i The number of PGs in the ith hard disk is the number of PGs to be stored in the unit hard disk.
4. The method for balanced storage of distributed storage according to claim 3, wherein the storage state data and the data to be stored are input into a balanced storage algorithm, the balanced storage algorithm in the balanced storage result is obtained and output, and the specific steps further include: receiving the linear programming operation result and analyzing and judging;
when the linear programming operation result is a non-integer, rounding the linear programming operation result to obtain the balanced storage result;
and when the linear programming operation result is an integer, taking the linear programming operation result as the balanced storage result.
5. The balanced storage method of a distributed storage according to any one of claims 1 to 4, wherein storing the data to be stored according to the balanced storage result specifically includes:
analyzing the balanced storage result to obtain a balanced storage strategy;
and storing the data to be stored according to the balanced storage strategy.
6. A distributed memory balancing storage apparatus, comprising: and a data acquisition module: acquiring data to be stored;
and a state reading module: reading the storage state of the distributed memory and outputting a storage state result; and the equalization algorithm module is used for: acquiring storage state data according to the storage state result; inputting the storage state data and the data to be stored into an equalization algorithm to obtain and output an equalization storage result; the equalization algorithm specifically comprises the steps of analyzing the data to be stored to obtain a data backup result to be stored; and calculating the number of PG to be stored of the unit hard disk according to the backup result of the data to be stored and the storage state data, and outputting a calculation result, wherein the number of PG to be stored of the unit hard disk is as follows:
PGs＝n×k/m
wherein PGs are the number of PG to be stored in a unit hard disk, n is the total storage space division number, k is the number of spare parts, and m is the total number of hard disks;
and the storage execution module is used for: and storing the data to be stored according to the balanced storage result.
7. A distributed memory balancing storage device, comprising: a memory, a processor, and a distributed memory balancing storage program stored on the memory and executable on the processor, the distributed memory balancing storage program configured to implement the distributed memory balancing storage method of any one of claims 1 to 5.
8. A computer readable storage medium storing a computer program, wherein the computer program is capable of implementing the steps of the distributed memory balancing storage method of any one of claims 1 to 5 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310717570.1A CN116737380B (en) | 2023-06-16 | 2023-06-16 | Balanced storage method, device and equipment for distributed memory and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310717570.1A CN116737380B (en) | 2023-06-16 | 2023-06-16 | Balanced storage method, device and equipment for distributed memory and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116737380A CN116737380A (en) | 2023-09-12 |
| CN116737380B true CN116737380B (en) | 2024-02-09 |
Family
ID=87916494
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310717570.1A Active CN116737380B (en) | 2023-06-16 | 2023-06-16 | Balanced storage method, device and equipment for distributed memory and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116737380B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106991170A (en) * | 2017-04-01 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of method and apparatus of distributed document capacity equilibrium |
| CN107317864A (en) * | 2017-06-29 | 2017-11-03 | 郑州云海信息技术有限公司 | The data balancing method and device of a kind of storage device |
| CN111064808A (en) * | 2019-12-30 | 2020-04-24 | 北京天融信网络安全技术有限公司 | Load balancing method and device based on distributed storage system |
| CN111562884A (en) * | 2020-04-28 | 2020-08-21 | 北京奇艺世纪科技有限公司 | Data storage method and device and electronic equipment |
| WO2022028033A1 (en) * | 2020-08-01 | 2022-02-10 | 广西大学 | Hierarchical mapping-based automatic balancing storage method for ceph storage system |
| CN114047883A (en) * | 2021-11-19 | 2022-02-15 | 北京天融信网络安全技术有限公司 | Data equalization method and device based on distributed storage system |
-
2023
- 2023-06-16 CN CN202310717570.1A patent/CN116737380B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106991170A (en) * | 2017-04-01 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of method and apparatus of distributed document capacity equilibrium |
| CN107317864A (en) * | 2017-06-29 | 2017-11-03 | 郑州云海信息技术有限公司 | The data balancing method and device of a kind of storage device |
| CN111064808A (en) * | 2019-12-30 | 2020-04-24 | 北京天融信网络安全技术有限公司 | Load balancing method and device based on distributed storage system |
| CN111562884A (en) * | 2020-04-28 | 2020-08-21 | 北京奇艺世纪科技有限公司 | Data storage method and device and electronic equipment |
| WO2022028033A1 (en) * | 2020-08-01 | 2022-02-10 | 广西大学 | Hierarchical mapping-based automatic balancing storage method for ceph storage system |
| CN114047883A (en) * | 2021-11-19 | 2022-02-15 | 北京天融信网络安全技术有限公司 | Data equalization method and device based on distributed storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116737380A (en) | 2023-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7169369B2 (en) | Method, system for generating data for machine learning algorithms | |
| US8849776B2 (en) | Method and system for resolving data inconsistency | |
| US7979520B2 (en) | Prescriptive architecture recommendations | |
| US11200511B1 (en) | Adaptive sampling of training data for machine learning models based on PAC-bayes analysis of risk bounds | |
| US20190034833A1 (en) | Model Training Method and Apparatus | |
| CN111414389A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111324422A (en) | Multi-target virtual machine deployment method, device, equipment and storage medium | |
| CN116257363A (en) | Resource scheduling method, device, equipment and storage medium | |
| CN110688055B (en) | Data access method and system in large graph calculation | |
| US20140181174A1 (en) | Distributed processing of stream data on an event protocol | |
| CN116737380B (en) | Balanced storage method, device and equipment for distributed memory and storage medium | |
| EP4276623A1 (en) | Sorting device and method | |
| CN114461384A (en) | Task execution method and device, computer equipment and storage medium | |
| CN117370488A (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN115858473B (en) | Data interaction method and device based on training system and object storage system | |
| CN111722923A (en) | Heterogeneous resource calling method and device and computer readable storage medium | |
| CN115442262B (en) | Resource evaluation method and device, electronic equipment and storage medium | |
| CN110909085A (en) | Data processing method, device, equipment and storage medium | |
| CN116647560A (en) | Method, device, equipment and medium for coordinated optimization control of Internet of things computer clusters | |
| CN114118381B (en) | Learning method, device, equipment and medium based on self-adaptive aggregation sparse communication | |
| CN113783919A (en) | Access request distribution method, system, device and storage medium | |
| CN113377866A (en) | Load balancing method and device for virtualized database proxy service | |
| US20230259365A1 (en) | Computer-readable recording medium storing conversion program and conversion method | |
| FI129343B (en) | Real-time process control with constrained resources | |
| CN115904717B (en) | Application system deployment method and device in cloud platform and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |