CN112286681A - Memory allocation management method, system, equipment and storage medium - Google Patents
Memory allocation management method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN112286681A CN112286681A CN202011159331.1A CN202011159331A CN112286681A CN 112286681 A CN112286681 A CN 112286681A CN 202011159331 A CN202011159331 A CN 202011159331A CN 112286681 A CN112286681 A CN 112286681A
- Authority
- CN
- China
- Prior art keywords
- memory
- data processing
- stage
- processing stage
- pool
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 230000015654 memory Effects 0.000 title claims abstract description 207
- 238000007726 management method Methods 0.000 title claims abstract description 36
- 238000004590 computer program Methods 0.000 claims description 10
- 230000003247 decreasing effect Effects 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 9
- 238000011084 recovery Methods 0.000 abstract description 5
- 230000004044 response Effects 0.000 abstract description 4
- 238000012544 monitoring process Methods 0.000 abstract description 3
- 230000009467 reduction Effects 0.000 abstract description 2
- 238000000638 solvent extraction Methods 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004064 recycling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
- G06F12/0253—Garbage collection, i.e. reclamation of unreferenced memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5022—Mechanisms to release resources
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Memory System (AREA)
Abstract
The invention discloses a memory allocation management method, a system, equipment and a storage medium. The method comprises the following steps: starting a system, applying for a memory by a storage controller, and setting the applied memory to be in an unrecoverable state; dividing the applied memory into a plurality of memory pools according to the data processing stage of the storage controller, wherein each memory pool corresponds to one stage of data processing; and dynamically adjusting the size of each memory pool according to the memory use condition in each memory pool. The system comprises a memory application module, a memory partitioning module and a memory management module. The invention fundamentally prevents the memory used for processing IO from being recovered by the operating system, avoids IO response delay and IO processing speed reduction possibly caused by memory recovery, also avoids the risk of failure recovery caused by the failure of collecting effective information in time by the monitoring module after storage failure, and ensures the collection of data recovery information during storage failure.
Description
Technical Field
The invention relates to the field of memory optimization of storage equipment, in particular to a memory allocation management method, a memory allocation management system, memory allocation management equipment and a storage medium.
Background
The storage controller is mainly used for managing a hard disk of the storage end and data stored on the hard disk, and providing data access service for the server end. In the process of providing data access to a server, because the number of Input/Output (IO) processed in a short time is huge, a memory in a storage controller is consumed in a large amount, and when the System memory pressure is large, an Operating System (OS) traverses the use condition of each process in the System and recovers a part of the memory, and meanwhile, after data in the memory is processed by a certain algorithm (such as a compression algorithm), the data is stored in a file, a hard disk or other places, which may cause two problems: if the memory for processing the IO thread is recycled, firstly, the IO processing speed of the storage controller and the IO response speed are influenced; secondly, if the storage node fails at this time, the monitoring process cannot capture and store the memory data in a short time, and functions such as later data recovery and failed node replacement are affected. The memory requirement of the thread processing IO cannot be met and guaranteed, and the situation that the application memory cannot be obtained due to insufficient memory may exist, which causes the delay of the whole IO processing process.
Disclosure of Invention
In order to solve the technical problems, the invention provides a memory allocation management method, a system, equipment and a storage medium, which optimize IO read-write speed, can reasonably utilize memory and ensure stable operation of a storage controller.
In order to achieve the purpose, the invention adopts the following technical scheme:
a memory allocation management method comprises the following steps:
starting a system, applying for a memory by a storage controller, and setting the applied memory to be in an unrecoverable state;
dividing the applied memory into a plurality of memory pools according to the data processing stage of the storage controller, wherein each memory pool corresponds to one stage of data processing;
and dynamically adjusting the size of each memory pool according to the memory use condition in each memory pool.
Further, the dividing the applied memory into a plurality of memory pools according to the stage of data processing of the storage controller, where each memory pool corresponds to one stage of data processing, includes:
and allocating the initial capacity of each memory pool according to the empirical value of the memory demand in each data processing stage.
Further, the method further comprises:
when data is written in, the current data processing stage is processed, if the next data processing stage runs idle, the access authority of the next data processing stage to the memory pool corresponding to the current data processing stage is opened;
and if the latter data processing stage is busy, transferring the data from the memory pool corresponding to the current data processing stage to the memory pool corresponding to the latter data processing stage, and releasing the memory pool corresponding to the current data processing stage.
Further, the method further comprises:
when reading data, processing of the current data processing stage is completed, if the next data processing stage runs idle, the access authority of the next data processing stage to the memory pool corresponding to the current data processing stage is opened;
and if the latter data processing stage is busy, transferring the data from the memory pool corresponding to the current data processing stage to the memory pool corresponding to the latter data processing stage, and releasing the memory pool corresponding to the current data processing stage.
Further, the memory pool is managed in a memory page manner.
Further, the dynamically adjusting the size of each memory pool according to the memory usage in each memory pool includes:
and periodically evaluating the use condition of the memory in the memory pool in each data processing stage, and increasing or decreasing the capacity of the memory pool according to the proportion of the used memory to the unused memory in each memory pool.
Further, the stage of data processing comprises: the system comprises a front-end interface layer stage, a cache stage, a storage pool stage, a disk array stage and a back-end interface layer stage.
The invention also provides a memory allocation management system, which comprises:
the memory application module is used for applying for a memory by the memory controller when the system is started and setting the applied memory into an unrecoverable state;
the memory dividing module is used for dividing the applied memory into a plurality of memory pools according to the data processing stage of the storage controller, and each memory pool corresponds to one stage of data processing;
and the memory management module is used for dynamically adjusting the size of each memory pool according to the memory use condition in each memory pool.
The invention also provides a device for memory allocation management, which comprises:
a memory for storing a computer program;
and the processor is used for realizing the steps of the memory allocation management method when executing the computer program.
The present invention also proposes a storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the memory allocation management method as described above.
The invention has the beneficial effects that:
the invention sets the allocated internal memory into an unrecoverable state by providing a method, a system, equipment and a storage medium for managing the internal memory allocation, thereby fundamentally putting an end to the situation that the internal memory for processing IO is recycled by an operating system, avoiding IO response delay and IO processing speed reduction possibly caused by the recycling of the internal memory, avoiding the risk that the fault cannot be recovered due to the fact that a monitoring module cannot collect effective information in time after the storage fault, and ensuring the collection of data recovery information when the storage fault occurs.
Meanwhile, the invention also realizes memory isolation, ensures the memory requirements of each data processing stage, selectively realizes memory inter-access, accelerates the IO processing flow, improves the performance of the storage controller and ensures the stability of the storage controller by reasonably and efficiently using the limited memory.
Drawings
FIG. 1 is a flow chart illustrating a memory allocation management method according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating a memory allocation flow of a memory allocation management method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a memory allocation management system according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a device for memory allocation management according to an embodiment of the present invention.
Detailed Description
In order to clearly explain the technical features of the present invention, the following detailed description of the present invention is provided with reference to the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different features of the invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and procedures are omitted so as to not unnecessarily limit the invention.
As shown in fig. 1, an embodiment of the present invention discloses a memory allocation management method, including:
starting a system, applying for a memory by a storage controller, and setting the applied memory to be in an unrecoverable state;
dividing the applied memory into a plurality of memory pools according to the data processing stage of the storage controller, wherein each memory pool corresponds to one stage of data processing;
and dynamically adjusting the size of each memory pool according to the memory use condition in each memory pool.
Specifically, the storage controller according to the embodiment of the present invention provides a data access service to an external server, and internally manages and organizes data on a back-end storage medium, so that a large amount of memory is required. At the initial stage of system startup, the storage controller applies for a large amount of memories from the operating system, and the applied memories are set to be in an unrecoverable state, so that the memories used for processing IO are fundamentally prevented from being recovered by the OS, IO response delay is avoided, and the risk that faults cannot be recovered is avoided.
As shown in fig. 2, the applied memory is managed in a memory page manner, corresponding memory pools 1 to 5 are respectively established in each data processing stage through which the corresponding data IO reaches from the server side after storage, and the initial capacity of each memory pool is allocated according to the empirical value of the memory demand in each data processing stage. Part of the stages of data processing are listed in fig. 2, including: a front-end interface layer stage, a cache stage, a storage pool stage, a disk array stage, a back-end interface layer stage, etc.
The memory pools of different data processing stages can selectively provide access interfaces for other data stages. When data is written in, the current data processing stage is processed, if the next data processing stage runs idle, the access authority of the next data processing stage to the memory pool corresponding to the current data processing stage is opened;
and if the latter data processing stage is busy, transferring the data from the memory pool corresponding to the current data processing stage to the memory pool corresponding to the latter data processing stage, and releasing the memory pool corresponding to the current data processing stage.
When reading data, processing of the current data processing stage is completed, if the next data processing stage runs idle, the access authority of the next data processing stage to the memory pool corresponding to the current data processing stage is opened;
and if the latter data processing stage is busy, transferring the data from the memory pool corresponding to the current data processing stage to the memory pool corresponding to the latter data processing stage, and releasing the memory pool corresponding to the current data processing stage.
For example, when data IO is written, the data comes to a VG-storage pool processing stage, if the operation of each stage is smooth and not busy, the permission is opened to the RAID layer for direct access, and the data is issued to the VL layer after the processing is finished; and if the RAID layer is busy and in a waiting state, the direct access of the RAID layer is not allowed, the data is stored in a memory pool corresponding to the RAID layer and waits for further processing, and the memory of the VG layer is released. And in the open condition of the access interface, two bits are used as flag bits which respectively represent that the memory pool is readable and writable.
The dynamically adjusting the size of each memory pool according to the memory usage in each memory pool includes: and periodically evaluating the use condition of the memory in the memory pool in each data processing stage, and increasing or decreasing the capacity of the memory pool according to the proportion of the used memory to the unused memory in each memory pool to maintain the memory pool in a reasonable proportion state.
As shown in fig. 3, an embodiment of the present invention further discloses a memory allocation management system, which includes:
the memory application module is used for applying for a memory by the memory controller when the system is started and setting the applied memory into an unrecoverable state;
the memory dividing module is used for dividing the applied memory into a plurality of memory pools according to the data processing stage of the storage controller, and each memory pool corresponds to one stage of data processing;
and the memory management module is used for dynamically adjusting the size of each memory pool according to the memory use condition in each memory pool.
As shown in fig. 4, an embodiment of the present invention further discloses a device for memory allocation management, including:
a memory for storing a computer program;
and the processor is used for realizing the steps of the memory allocation management method when executing the computer program.
The embodiment of the invention also discloses a storage medium, wherein a computer program is stored on the storage medium, and the steps of the memory allocation management method are realized when the computer program is executed by a processor.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, the scope of the present invention is not limited thereto. Various modifications and alterations will occur to those skilled in the art based on the foregoing description. And are neither required nor exhaustive of all embodiments. On the basis of the technical scheme of the invention, various modifications or changes which can be made by a person skilled in the art without creative efforts are still within the protection scope of the invention.
Claims (10)
1. A memory allocation management method is characterized by comprising the following steps:
starting a system, applying for a memory by a storage controller, and setting the applied memory to be in an unrecoverable state;
dividing the applied memory into a plurality of memory pools according to the data processing stage of the storage controller, wherein each memory pool corresponds to one stage of data processing;
and dynamically adjusting the size of each memory pool according to the memory use condition in each memory pool.
2. The memory allocation management method according to claim 1, wherein the dividing the applied memory into a plurality of memory pools according to the stage of data processing of the storage controller, each memory pool corresponding to one stage of data processing includes:
and allocating the initial capacity of each memory pool according to the empirical value of the memory demand in each data processing stage.
3. The memory allocation management method according to claim 1, further comprising:
when data is written in, the current data processing stage is processed, if the next data processing stage runs idle, the access authority of the next data processing stage to the memory pool of the current data processing stage is opened;
and if the latter data processing stage is busy, storing the data from the memory pool of the current data processing stage into the memory pool of the latter data processing stage, and releasing the memory pool of the current data processing stage.
4. The memory allocation management method according to claim 1, further comprising:
when reading data, processing of the current data processing stage is completed, if the next data processing stage runs idle, the access authority of the next data processing stage to the memory pool of the current data processing stage is opened;
and if the latter data processing stage is busy, storing the data from the memory pool of the current data processing stage into the memory pool of the latter data processing stage, and releasing the memory pool of the current data processing stage.
5. The memory allocation management method according to claim 1, wherein the memory pool is managed in a memory page manner.
6. The memory allocation management method according to claim 1, wherein the dynamically adjusting the size of each memory pool according to the memory usage in each memory pool comprises:
and periodically evaluating the use condition of the memory in the memory pool in each data processing stage, and increasing or decreasing the capacity of the memory pool according to the proportion of the used memory to the unused memory in each memory pool.
7. The memory allocation management method according to claim 1, wherein the data processing stage comprises: the system comprises a front-end interface layer stage, a cache stage, a storage pool stage, a disk array stage and a back-end interface layer stage.
8. A memory allocation management system, comprising:
the memory application module is used for applying for a memory by the memory controller when the system is started and setting the applied memory into an unrecoverable state;
the memory dividing module is used for dividing the applied memory into a plurality of memory pools according to the data processing stage of the storage controller, and each memory pool corresponds to one stage of data processing;
and the memory management module is used for dynamically adjusting the size of each memory pool according to the memory use condition in each memory pool.
9. An apparatus for memory allocation management, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the memory allocation management method according to any one of claims 1 to 7 when executing the computer program.
10. A storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the memory allocation management method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011159331.1A CN112286681A (en) | 2020-10-26 | 2020-10-26 | Memory allocation management method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011159331.1A CN112286681A (en) | 2020-10-26 | 2020-10-26 | Memory allocation management method, system, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112286681A true CN112286681A (en) | 2021-01-29 |
Family
ID=74374139
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011159331.1A Withdrawn CN112286681A (en) | 2020-10-26 | 2020-10-26 | Memory allocation management method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112286681A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292052A (en) * | 2022-09-27 | 2022-11-04 | 荣耀终端有限公司 | Memory recovery method, electronic device and computer-readable storage medium |
-
2020
- 2020-10-26 CN CN202011159331.1A patent/CN112286681A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292052A (en) * | 2022-09-27 | 2022-11-04 | 荣耀终端有限公司 | Memory recovery method, electronic device and computer-readable storage medium |
| CN115292052B (en) * | 2022-09-27 | 2023-08-08 | 荣耀终端有限公司 | Memory recycling method, electronic device and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| DE102015014851B4 (en) | Allocation and release of resources for energy management in devices | |
| US9946655B2 (en) | Storage system and storage control method | |
| KR102093523B1 (en) | Working set swapping using a sequentially ordered swap file | |
| CN104407933B (en) | A kind of backup method and device of data | |
| WO2022017002A1 (en) | Garbage collection method and device | |
| CN106066890B (en) | Distributed high-performance database all-in-one machine system | |
| JP2019079113A (en) | Storage device, data management method, and data management program | |
| CN102902602B (en) | The method of data thermal backup, device and storage system | |
| CN103902475B (en) | Solid state disk concurrent access method and device based on queue management mechanism | |
| DE102021115626A1 (en) | DATA AGGREGATION IN CNS DRIVE | |
| CN103514249A (en) | Method and system for automatic data reduction and storage device | |
| JP2012234254A (en) | Memory system | |
| WO2020007030A1 (en) | System controller and system garbage recovery method | |
| CN108255630A (en) | A kind of method for reducing solid state disk powered-off fault processing time | |
| CN109918318A (en) | SSD metadata management method, device, equipment and readable storage medium storing program for executing | |
| CN115167782B (en) | Temporary storage copy management method, system, equipment and storage medium | |
| CN102025758B (en) | Method, device and system for recovering data copy in distributed system | |
| CN112286681A (en) | Memory allocation management method, system, equipment and storage medium | |
| CN106469119A (en) | A kind of data write buffer method based on NVDIMM and its device | |
| US20210081321A1 (en) | Method and apparatus for performing pipeline-based accessing management in a storage server | |
| US20130246710A1 (en) | Storage system and data management method | |
| US10360990B2 (en) | Information processing device, information processing method, and program | |
| DE112020005096T5 (en) | WEIGHTING OF READ COMMANDS TO ZONES IN STORAGE DEVICES | |
| US11055184B2 (en) | In-place garbage collection of a sharded, replicated distributed state machine based on supersedable operations | |
| CN111338846B (en) | Method and device for recovering L2P (L2P) table based on multiple cores |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210129 |
|
| WW01 | Invention patent application withdrawn after publication |