CN118245396A

CN118245396A - Exchange memory optimization method based on asynchronous deduplication and recompression

Info

Publication number: CN118245396A
Application number: CN202410091063.6A
Authority: CN
Inventors: 熊浩宇; 孙韬宁; 龙家璇; 杜小华; 曹学贵
Original assignee: Chongqing University
Current assignee: Chongqing University
Priority date: 2024-01-23
Filing date: 2024-01-23
Publication date: 2024-06-25

Abstract

The application provides an exchange memory optimization method based on asynchronous deduplication and recompression, relates to the technical field of computers, and solves the problems that the amount of storage space is small because compressed data continuously fed into ZRAM space occupies a large amount of storage space when data in a memory is gradually increased under the conditions that memory resources of mobile phone equipment are limited and applications opened by users are more. The method comprises the following steps: starting an independent thread in the ZRAM space to scan, and sending the page with the retention time exceeding the threshold value 1 to FLASH; for pages exceeding the threshold 2 but being smaller than the threshold 1, searching for repeated pages with a plurality of pages_crc, decompressing the repeated pages and comparing the contents of the pages to realize judgment of a page sharing mechanism; other pages will perform a recompression operation in that interval using an algorithm that is better in compression performance.

Description

Exchange memory optimization method based on asynchronous deduplication and recompression

Technical Field

The application relates to the technical field of computers, in particular to a switching memory optimization method based on asynchronous deduplication and recompression.

Background

With the popularity of mobile devices, there is a growing demand for applications and functions on mobile platforms. Particularly, under the condition that multiple applications run simultaneously, the mobile system faces challenges of competing and managing memory resources among different applications, and efficient memory allocation and recovery are required to be realized. Although memory management can be improved by adding physical memory, this is not a long-term solution due to cost and scalability issues. Therefore, in order to fully utilize limited memory resources to the greatest extent to meet the functional requirements of various applications, current research focuses on optimizing the existing memory management scheme without introducing additional space overhead, reducing unnecessary memory waste, and improving overall system performance.

In order to use memory more efficiently, developers of the Linux community and large mobile device manufacturers have proposed many useful features and functional components to alleviate the problem of memory starvation. In the user space level, software developers realize the application design with maximized memory benefit; in the kernel layer of the system, a system developer needs to effectively organize the memory space and reasonably allocate and release the resource requirements of the upper layer. In view of the challenges presented above, the compressed memory switching technology, i.e., ZRAM technology, has become a key innovative technology for dealing with limited memory resources and multiple application requirements. The compressed memory swapping technique reduces the data volume by compressing memory data to be swapped to the auxiliary memory, and then swaps the compressed data to the auxiliary memory. When the data needs to be reloaded, decompression operation is carried out, and the data is restored to the memory.

However, after extensive analysis of page behavior in ZRAM space using trace tracking tools in Linux, a significant proportion of duplicate pages were observed in pages moved into ZRAM space. For page duplication situations that occur when pages are fed into ZRAM space, some research and solutions focus on real-time deduplication techniques to optimize ZRAM space utilization. Further research shows that when the page in the ZRAM space stays, only about 10% -20% of compressed pages are observed to be replaced into the memory for reuse. This means that a large number of cold pages exist in ZRAM space, which are not frequently used, but occupy valuable memory resources. Therefore, how to solve the above-mentioned problems and improve the space utilization of ZRAM becomes a technical problem in the current research.

Disclosure of Invention

Therefore, the present study is directed to the above problems, and is directed to the situation of limited resources in mobile devices, and aims to propose a good memory optimization method to improve the performance and memory utilization of ZRAM space. By proposing innovative optimization strategies to address these challenges, research is directed to improving ZRAM performance on mobile devices, reducing the presence of cold pages, and optimizing the allocation of memory resources to better accommodate the specific needs of the mobile device.

In order to achieve the above purpose, the invention constructs a memory space expansion optimization method under the condition of limited mobile equipment resources.

The memory space expansion optimization method modifies the ZRAM space management mechanism to support page migration from the ZRAM to the FLASH, and designs an asynchronous deduplication mechanism and a page recompression mechanism in the ZRAM space.

The application provides an innovative memory space expansion optimization method, which is particularly suitable for the situation of limited resources such as mobile equipment and the like, and comprises the following steps:

The modification of the ZRAM space management mechanism realizes the support of page migration between FLASH and ZRAM, thereby improving the utilization efficiency of the memory. While detailed recording of page information is introduced for more efficient management of ZRAM space. These records include a unique identification for each page, a compressed size, a reference, and a reference count. This approach enables the system to more intelligently handle the compression and migration requirements of different pages.

In the process of page swap-in ZRAM space, a faster compression algorithm is adopted to ensure efficient and rapid page swap-in under the environment of limited resources. Meanwhile, in order to periodically optimize the utilization of ZRAM space, an independent thread is introduced, and pages stored in the ZRAM space are periodically scanned and compressed.

In order to further improve the utilization efficiency of the memory, a cold and hot judging mechanism of the page is introduced. Pages that exceed the set threshold 1 are considered cold pages and periodically migrate to FLASH space. During the scanning process, for the page between the threshold 1 and the threshold 2, the recompression operation and the page sharing mechanism judgment are performed. For pages exceeding the threshold 2 but being smaller than the threshold 1, the system searches for repeated pages with a plurality of pages_crc, performs decompression and content comparison operations, realizes page sharing and correspondingly increases the shared page count. And for other pages in the interval, recompression is carried out by adopting a more optimized compression algorithm so as to optimize the using effect of the ZRAM space to the greatest extent.

Drawings

FIG. 1 is a flowchart of an optimization method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of a Linux trace buried point provided in an embodiment of the present application;

FIG. 3 is a statistical plot of refault time distributions of pages provided in an embodiment of the present application;

FIG. 4 is a diagram of a memory space expansion optimization framework provided in an embodiment of the present application;

fig. 5 is a flowchart of a memory space expansion optimization framework provided in an embodiment of the present application.

Detailed Description

The following describes the embodiments and working principles of the present invention in further detail with reference to the drawings.

The invention aims at providing a set of memory space expansion optimization method under the condition of limited mobile equipment resources. Because mobile phone equipment resources are limited and the demands of users for memory are increasing, smooth switching of a plurality of applications cannot be realized. In order to realize efficient and high-resource-utilization-rate memory use, smooth use experience of users is provided. FIG. 1 illustrates a memory management optimization method, wherein pages are scanned by independent threads to evaluate their page cold and hot states after they are compressed into ZRAM space. Pages identified as cold will be migrated to FLASH storage, while the system will make recompression or sharing mechanism decisions for pages that stay between two preset thresholds.

The present invention mainly relates to two key aspects of work, firstly, the present work explores and analyzes migration information of pages in detail. Secondly, based on the page behavior information, a memory space expansion optimization method under the condition of limited mobile equipment resources is provided.

In the aspect of page behavior exploration, trace tracking tools in Linux are used for carrying out buried point tracking on migration behaviors of pages, and key information such as scheduling behaviors and page stay between a memory and a ZRAM space is captured. Specifically, a large continuous space is actively allocated in the memory, and is specially used for storing trace information. Simultaneously, three related files are created under the/proc directory, so that the opened buffer space can be controlled better.

First, an action_switch file is introduced, whose role is to control whether behavior statistics are to be started. By setting the file, a set may be selected to turn on statistics, and a clear indicates turn off statistics. Next, an action_ bufsize file is designed for controlling the number of data pieces written into an action. Once the number of data pieces exceeds the set point, the system resets statistics, ensuring a timely and efficient record. And finally, creating an action file for actually recording trace information obtained from statistics. This file carries the essential content of the trace data, providing the necessary material for subsequent analysis.

FIG. 2 is a schematic diagram of a buried point tracking framework, in which ADD trace buried points are executed when a new page is added to the LRU linked list, and the adding action of the page is recorded at the moment; executing a MOVE trace embedded point for migration behaviors of active and inactive LRU linked list pages, and recording an action of migrating a page from one linked list to another linked list; when the LRU linked list ISOLATEs the page to be operated, an ISOLATE trace embedded point is executed, and at the moment, the isolation operation is recorded; when the pages are released and recycled in the LRU linked list, executing RELEASE and RECLAIM TRACE buried points, and recording specific actions of page RELEASE and recycling; finally, when the anonymous page is swapped IN and swapped OUT, SWAP OUT and SWAP IN TRACE buried points are executed, and actions of swapping the anonymous page OUT of the memory to the ZRAM space (SWAP OUT) and swapping the anonymous page into the memory (SWAP IN) from the ZRAM space are recorded. The execution of the trace embedded point operations enables the system to record various migration behaviors of the page in detail, including the access mode, state, compression information, position, migration time and other information of the page, provides accurate and comprehensive information for further page behavior analysis, and is beneficial to understanding and optimizing the memory management behavior of the system.

Next, a compressed memory exchange scheme in the prior art is described.

The current compressed memory exchange scheme reduces the data volume by compressing memory data to be exchanged to the auxiliary memory, and then exchanges the compressed data to the auxiliary memory. When the data needs to be reloaded, decompression operation is carried out, and the data is restored to the memory.

From the above process, after analysis by the buried point tracking framework, it is known that there are a considerable proportion of repeated pages in the pages moved into the ZRAM space, the number of the repeated pages is compared with that of table 1, and these repeated pages occupy the ZRAM space, so that the effective utilization rate of the memory is reduced.

TABLE 1

Data	Number of repeated pages	Swap out page quantity	Repeated page duty cycle
				Test1	61106	291111	21％
Test2	55780	306376	18.2％
				Test3	46208	274204	16.9％

Further research shows that when the stay condition of the pages IN the ZRAM space is found, only about 10% -20% of compressed pages are observed to be replaced into the memory for reuse, and the SWAP IN times IN the ZRAM space are compared with the reference table 2. FIG. 2 is a statistical plot of the refault time distribution of a page, which can be found from the inflection points in the plot, the longer the residence time in the ZRAM space, the less likely it is to be to swap back into memory. The above experiments all mean that a large number of cold pages exist in ZRAM space, which are not frequently used, but occupy valuable memory resources.

TABLE 2

Test case	Number of SWAP IN	Number of SWAP OUT times	IN/OUT
				First group of	53906	358191	15％
Second group of	28584	315799	9％
				Third group of	33849	294602	11％
Fourth group	51401	291111	18％
				Fifth group of	19329	306376	6％
Sixth group of	39239	274204	14％

In view of this, the embodiment of the application provides a memory space expansion optimization method, which aims at increasing the memory capacity of a mobile phone by expanding the ZRAM space on the premise of not affecting user experience, and reducing page I/O operation to improve the system performance.

The following describes the technical scheme provided by the embodiment of the application with reference to the accompanying drawings.

The embodiment of the application provides a memory expansion optimization method, and fig. 4 is a frame diagram of the method. The left side of fig. 4 is a normal memory space, and the right side is a compressed memory space, i.e. a ZRAM space, and the optimization method mainly relates to cold and hot identification and management of pages in the ZRAM space. In the processing framework shown in fig. 4, the system scans ZRAM through independent threads, considers pages with dwell times exceeding a set threshold as cold pages, and migrates to FLASH storage. And meanwhile, content comparison is carried out on pages with the same page_crc, so that page deduplication is realized. In addition, a recompression operation is performed on a part of pages meeting the threshold condition by using a recompressor, so as to further optimize the memory use efficiency and the system performance.

FIG. 5 is a flow chart of the framework detailing a system method for optimizing ZRAM space usage. First, the framework enables it to support page migration between FLASH and ZRAM simultaneously by modifying the management mechanism of the ZRAM space. For the page to be swapped into the ZRAM space, if the page is checked as the zero-filled page, the page is directly marked as zero-FILLED PAGE, so that the compression and storage of the page are avoided, and precious CPU cycles are saved. For other non-zero-filled pages, a faster compression algorithm is used directly to compress and swap it into ZRAM space. During the swap-in process, each page will be attached with a new data structure that will contain the following information: 1) Page_crc: a memory page can be uniquely identified; 2) size: page compression size; 3) ref: a reference to the compressed page; 4) ref_count: the number of this page is shared.

Secondly, the optimization work of the invention is mainly aimed at compressing pages in the ZRAM space. For pages that have been stored in ZRAM space, an independent thread will be actively started for the time to scan the compressed pages. If the page dwell time exceeds the set threshold value 1, it will be determined that such pages are cold pages and immediately migrate to FLASH.

Then, during the scan, if the time of the page falls between threshold 1 and threshold 2, a determination of the recompression and page sharing mechanism will be performed for pages within such time interval. For those pages that exceed threshold 2 but are less than threshold 1, pages with multiple Page_crcs are sought, i.e., if Page-A and Page-B have the same Page_crc, then it is considered that such pages may be the same duplicate Page, where Page_crc can uniquely identify the Page. At the same time, the system decompresses the pages in the section, and the page sharing mechanism compares the content of the pages in the section with the content of the pages in the section. If the contents are consistent, the pages share the same page, and the ref_count count is increased accordingly.

Finally, if no page of the same page_crc information is found, other pages of such an interval will also be of interest. The recompression mechanism dynamically selects and adjusts the most appropriate compression algorithm to execute the recompression operation on the pages according to the page content, structure and historical compression conditions so as to further optimize the use effect of the ZRAM space.

Claims

1. A memory space expansion optimization framework in a mobile device resource constrained situation, the framework comprising:

Modifying the management mechanism to enable the management mechanism to support page migration between ZRAN and FLASH;

introducing a data structure, and recording page information, wherein the page information comprises unique identification, compression size, reference count and the like;

And starting an independent thread to process the page stored in the ZRAM space, and respectively executing page migration operation, page sharing mechanism judgment and recompression operation on the compressed page according to the threshold value.

2. The method of claim 1, wherein the management mechanism migrates a cold page into ZRAM space to FLASH, comprising:

Periodically scanning compressed pages entering the ZRAM space according to the independent threads, and recording the existence time of the compressed pages;

if the existence time of the compressed page exceeds the set threshold value, the page is considered to be a cold page existing in the ZRAM space;

the management mechanism triggers the page migration operation to migrate the cold pages to FLASH.

3. The method of claim 1, wherein the page sharing mechanism is to asynchronously deduplicate duplicate pages that are entered into ZRAM space, comprising:

According to the related information recorded by the data structure, the sharing mechanism can search pages with a plurality of unique identifications;

If a plurality of pages with the same unique identification can be found, the optimization framework will decompress the pages, and the page sharing mechanism will compare the content of the pages in detail;

If the page contents are consistent, the pages will share the same page.

4. The method of claim 1, wherein the recompression operation recompresses pages into ZRAM space using a better compression algorithm, comprising:

finding page intervals with preset thresholds according to the independent threads, wherein the optimization framework can decompress pages in the intervals;

The recompression mechanism dynamically selects and adjusts the most suitable compression algorithm according to the content, the structure and the historical compression condition of the page, and the recompression operation is carried out on the whole page of the section.