CN113296940A

CN113296940A - Data processing method and device

Info

Publication number: CN113296940A
Application number: CN202110362007.8A
Authority: CN
Inventors: 朱延海; 任镇
Original assignee: Alibaba Singapore Holdings Pte Ltd
Current assignee: Alibaba Innovation Co
Priority date: 2021-03-31
Filing date: 2021-04-02
Publication date: 2021-08-24
Anticipated expiration: 2041-04-02
Also published as: CN113296940B

Abstract

An embodiment of the present specification provides a data processing method and an apparatus, wherein the data processing method includes: receiving a data processing request aiming at a memory page, and acquiring and scanning a memory page list based on the data processing request; determining at least one memory page to be compressed from the memory page list according to a preset algorithm, wherein each memory page to be compressed comprises a first page descriptor, a second page descriptor and at least one third page descriptor; creating a data management structure for the memory page to be compressed based on the first page descriptor, the second page descriptor, and at least one third page descriptor; copying the first page descriptor and the second page descriptor from the memory page to be compressed, storing the copied memory page and the second page descriptor in the data management structure, and deleting the first page descriptor, the second page descriptor and at least one third page descriptor of the memory page to be compressed.

Description

Data processing method and device

Technical Field

The embodiment of the specification relates to the technical field of computers, in particular to two data processing methods. One or more embodiments of the present specification relate to both data processing apparatuses, a computing device, and a computer-readable storage medium.

Background

With the rapid development of big data, mass data are produced, so that the requirement on a data storage space is increased, and based on the requirement, the current processing mode is to compress memory data, one is to compress the memory data and store the compressed memory data to an external memory to vacate more physical memory, and the other is to select a cold memory page frame and compress the data in the page frame to increase the idle memory, so that the memory data is processed, not only more computing resources are consumed, but also the access of a user is delayed, and the access efficiency is not high.

Disclosure of Invention

In view of this, the embodiments of the present specification provide two data processing methods. One or more embodiments of the present specification relate to two data processing apparatuses, a computing device, and a computer-readable storage medium, so as to solve technical shortcomings in the prior art.

According to a first aspect of embodiments herein, there is provided a data processing method including:

receiving a data processing request aiming at a memory page, and acquiring and scanning a memory page list based on the data processing request;

determining at least one memory page to be compressed from the memory page list according to a preset algorithm, wherein each memory page to be compressed comprises a first page descriptor, a second page descriptor and at least one third page descriptor;

creating a data management structure for the memory page to be compressed based on the first page descriptor, the second page descriptor, and at least one third page descriptor;

copying the first page descriptor and the second page descriptor from the memory page to be compressed, storing the copied memory page and the second page descriptor in the data management structure, and deleting the first page descriptor, the second page descriptor and at least one third page descriptor of the memory page to be compressed.

According to a second aspect of embodiments herein, there is provided another data processing method including:

receiving a data processing request aiming at a memory page, and distributing a data storage structure for the memory page based on the data processing request;

obtaining a page table corresponding to the memory page, and determining a structure address of a data management structure corresponding to the memory page based on the page table;

obtaining a first page descriptor and a second page descriptor of the memory page from the data management structure based on the structure address, and determining at least one third page descriptor based on the second page descriptor;

storing the first page descriptor, the second page descriptor, and at least one third page descriptor to the data storage structure.

According to a third aspect of embodiments herein, there is provided a data processing apparatus comprising:

a first receiving module, configured to receive a data processing request for a memory page, and obtain and scan a memory page list based on the data processing request;

a first determining module, configured to determine at least one memory page to be compressed from the memory page list according to a preset algorithm, where each memory page to be compressed includes a first page descriptor, a second page descriptor, and at least one third page descriptor;

a creation module configured to create a data management structure for the memory page to be compressed based on the first page descriptor, the second page descriptor, and at least one third page descriptor;

a first storage module configured to copy the first page descriptor and the second page descriptor from the memory page to be compressed, store the copied memory page and the second page descriptor in the data management structure, and delete the first page descriptor, the second page descriptor, and at least one third page descriptor of the memory page to be compressed.

According to a fourth aspect of embodiments herein, there is provided another data processing apparatus including:

a second receiving module, configured to receive a data processing request for a memory page, and allocate a data storage structure for the memory page based on the data processing request;

a second determining module, configured to acquire a page table corresponding to the memory page, and determine, based on the page table, a structure address of a data management structure corresponding to the memory page;

a fetching module configured to fetch a first page descriptor and a second page descriptor of the memory page from the data management structure based on the fabric address, and determine at least one third page descriptor based on the second page descriptor;

a second storage module configured to store the first page descriptor, the second page descriptor, and at least one third page descriptor to the data storage structure.

According to a fifth aspect of embodiments herein, there is provided a computing device comprising:

a memory and a processor;

the memory is used for storing computer-executable instructions, and the processor is used for executing the computer-executable instructions to realize the steps of the data processing method.

According to a sixth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of any one of the data processing methods.

One embodiment of the present specification obtains and scans a memory page list based on a data processing request received for a memory page; determining at least one memory page to be compressed from the memory page list according to a preset algorithm, wherein each memory page to be compressed comprises a first page descriptor, a second page descriptor and at least one third page descriptor; creating a data management structure for the memory page to be compressed based on the first page descriptor, the second page descriptor, and at least one third page descriptor; the first page descriptor and the second page descriptor are copied from the memory page to be compressed and stored in the data management structure, the first page descriptor, the second page descriptor and at least one third page descriptor of the memory page to be compressed are deleted, the first page descriptor and the second page descriptor are copied and stored in the data management structure, the page descriptor of the memory page is released, the page descriptor of the memory page is compressed, the memory page can be saved, the memory space can be saved, more CPU computing resources can be further saved, the computing pressure of a CPU can be reduced, and the data access efficiency can be improved when a user accesses data.

Drawings

Fig. 1 is a schematic system structure diagram of an application of a data processing method provided in an embodiment of the present specification;

FIG. 2 is a flow chart of a data processing method provided by an embodiment of the present specification;

FIG. 3 is a diagram illustrating a data management structure of a data processing method according to an embodiment of the present disclosure;

FIG. 4 is a flow diagram of another data processing method provided by one embodiment of the present description;

FIG. 5 is a flowchart illustrating a data processing method according to an embodiment of the present disclosure;

FIG. 6 is a flow chart of another data processing method provided by an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present specification;

FIG. 8 is a block diagram of another data processing apparatus according to an embodiment of the present disclosure;

fig. 9 is a block diagram of a computing device according to an embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.

The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

First, the noun terms to which one or more embodiments of the present specification relate are explained.

Page frame or page: also called page, is the basic unit of operating system memory management, with a standard page size of 4 KB.

And (4) large page: the page is also called hugepage, which is a memory block with 2MB as a unit, and compared with a standard page of 4KB, the page is often used in scenarios such as virtualization and databases to improve performance.

A page descriptor: also called struct page, is a structure for managing page frame status information.

Page table: also called the page table, is a data structure that translates process-visible virtual addresses to physical addresses.

Page table entry: also called PTE (Page Table entry), 64 bits, where bits 0-11 are PTE flags, 12-MAXPHYLADDR is the physical address, MAXPHYLADDR is the maximum address width, relevant to the system setup.

And (4) missing page processing program: the mechanism for allocating memory on demand triggers the process of allocating memory when the address has not already allocated memory.

Compression: this is referred to herein as the process of freeing the remaining page descriptors after certain information is computed and saved, and not the compression algorithm in general.

Decompressing: the reverse of "compaction," reallocating and populating the descriptor as well, restores the original large page data structure state.

The Linux operating system manages a 4KB page frame with a 64B sized page description (struct page) with an overhead of 1/64. When the server memory is large, the management overhead is considerable, for example 384G, 6GB memory. Currently, a cloud computing server generally allocates memory to a virtual machine according to large pages, one large page includes 512 standard pages, and 512 page descriptors are correspondingly consumed, but only a first page descriptor (called a head page) is enough to describe and track a final state of the large page, and the remaining page descriptors are either in an idle state or can be derived from the first page descriptor. Therefore, the transparent compression method for the page descriptors is provided aiming at the characteristic, free page descriptors can be released, memory management cost is reduced, and memory of a physical server can be effectively saved by reducing the cost of a large-page memory management structure of an operating system, for example, about 5G of memory can be saved for a server with 384G of memory.

There may be two ways of memory compression: firstly, compressing a general memory, namely compressing memory data, then replacing the compressed memory data to an external memory, and vacating more physical memories; the defects are that CPU is consumed, and memory is replaced very slowly when page faults occur. Secondly, compressing the large-page memory, selecting a cold memory page frame, compressing data in the page frame, and increasing an idle memory; the defects are that CPU is consumed and memory access delay is increased.

It can be seen that the above methods compress the data stored in the page frame, and the present application compresses the page frame descriptor.

In the present specification, two data processing methods are provided, and the present specification relates to two data processing apparatuses, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

Fig. 1 is a schematic system structure diagram illustrating an application of a data processing method according to an embodiment of the present specification.

It should be noted that, in the present application, the page descriptor is compressed by using the characteristics of the memory page management structure, the compression process does not use the compression algorithm consuming the CPU, but directly releases the memory during compression, and recovers the information in the page descriptor in a copy manner during decompression, so that a considerable memory can be saved in the physical service with a larger memory, for example, a 384G memory, which can save nearly 5G memory.

Fig. 1 a is a page descriptor compression controller, where the compression controller includes a compression unit a1, a decompression unit a2, a memory care process module, and a page-missing processing program module, where the memory care process triggers the compression unit to work when the memory care process determines that the current memory is in shortage, the compression unit starts scanning a memory page list, picks out a memory page to be compressed, copies a first head page descriptor and a first tail page descriptor in page descriptors of the memory page to be compressed, stores the first head page descriptor and the first tail page descriptor in a data storage structure, stores an address of the data storage structure in a page table, and releases the page descriptors to compress the page descriptors of the memory page.

When the page descriptors of one memory page are compressed, the identification of the page table entries needs to be cleared, so that when the kernel needs to access the page descriptors, the kernel can be trapped in a missing page processing program, a decompression unit is triggered to work, the first page descriptors and the second page descriptors are obtained according to the stored structure address in the page table, the third page descriptors are determined according to the second page descriptors, the third page descriptors are distributed, and the page descriptors of the memory page are restored to the state before compression.

The data processing method provided in the embodiments of the present description proposes a method for compressing and decompressing page descriptors of a memory page according to the characteristic that page descriptors of a memory page management structure other than a header page descriptor are idle and can be recycled for other use, and the method can be operated and used economically and efficiently to reduce memory management overhead.

Referring to fig. 2, fig. 2 shows a flowchart of a data processing method provided in an embodiment of the present specification, which specifically includes the following steps:

step 202: receiving a data processing request aiming at a memory page, and acquiring and scanning a memory page list based on the data processing request.

Wherein, the data processing request can be understood as a data compression request of a user for the page descriptor; a memory page list may be understood as a list of memory pages that store memory data.

Specifically, the compression unit of the page descriptor compression controller receives a data processing request for a memory page, and acquires and scans a memory page list based on the data processing request.

In practical applications, before receiving the data processing request for the memory page, the method further includes:

acquiring a target processing page number aiming at a current memory page under the condition that a memory monitoring system determines that a memory value of the current memory page meets a preset memory condition is received;

and determining the current processing page number of the current memory page, and receiving a data processing request aiming at the current memory page under the condition that the current processing page number is less than the target processing page number.

Specifically, when receiving a condition that the memory monitoring system determines that the memory value of the current memory page meets a preset memory condition, the compression unit of the page descriptor compression controller acquires a target processing page number of the current memory page, determines a current processing page number of the current memory page, and receives a data processing request for the current memory page number when the current processing page number is smaller than the target processing page number.

In practical application, a kernel of a memory monitoring system can wake up and check the state of a system memory periodically through a memory nursing process, the memory value of a system memory page can be monitored in real time, when the available memory is lower than the water level, namely the memory value meets a preset memory condition and exceeds a preset memory threshold value, the system memory is indicated to be in shortage so as to trigger a compression mechanism to work, measures such as recovery cache and the like are tried to release more available memory, a compression unit starts a compression process, then, for a target processing page number, which can be input by a system administrator in advance, different target processing page numbers can be set according to different system memory spaces, and the specification does not limit the target processing page number; further, after the target processing page number of the current memory page is obtained, the compressing unit may determine the current processing page number of the current memory page, and when the current processing page number is smaller than the target processing page number, it indicates that the compressing unit needs to continuously compress the page descriptors of the memory page until the current processing page number reaches the target processing page number.

For example, the memory monitoring system determines that the current system memory value is 384G, when the memory monitoring system determines that the remaining free memory is 4G and is lower than the system memory level, the compression unit is triggered to operate, and further determines that the target processing page number is 5 pages, and the current processing page number is 0, the compression unit operates to start a compression process, and compresses the page descriptors of the memory pages until the memory page number of the compressed page descriptors reaches 5, and the compression unit stops the compression operation.

Further, the obtaining a target processing page number for the current memory page when it is received that the memory monitoring system determines that the memory value of the current memory page meets a preset memory condition includes:

starting a using interface when a condition that a memory monitoring system determines that a memory value of a current memory page meets a preset memory condition is received;

acquiring a target processing page number aiming at the current memory page based on the using interface;

correspondingly, receiving a data processing request for the current memory page when the current processing page number is smaller than the target processing page number includes:

receiving a data processing request for the current memory page based on a usage interface if the current number of processed pages is less than the target number of processed pages.

Specifically, after triggering the compression unit to start the compression process, the memory monitoring system starts the use interface, obtains a target processing page number of the current memory page based on the use interface, and receives a data processing request of the current memory page through the started use interface under the condition that the current processing page number is smaller than the target processing page number.

The embodiment of the specification realizes the acquisition of data by starting an enabling switch and utilizing an interface to realize the triggering of the compressed page descriptor processing.

Step 204: and determining at least one memory page to be compressed from the memory page list according to a preset algorithm, wherein each memory page to be compressed comprises a first page descriptor, a second page descriptor and at least one third page descriptor.

The preset algorithm may be understood as an algorithm for screening memory pages to be compressed from the memory page list, such as a clock-like algorithm.

Specifically, the compression unit starts scanning the memory page list, preferentially scans the free page list, and secondly scans the active list, and each memory page to be compressed includes a first page descriptor, a second page descriptor, and at least one third page descriptor.

In practical applications, the memory page list includes a first type memory page list and a second type memory page list;

correspondingly, the determining at least one to-be-compressed memory page from the memory page list according to a preset algorithm includes:

under the condition that the first type memory page list comprises at least one memory page to be compressed according to a preset algorithm, acquiring the at least one memory page to be compressed; or

Under the condition that the second type memory page list comprises at least one memory page to be compressed according to a preset algorithm, acquiring the at least one memory page to be compressed; or

And under the condition that it is determined that the first type memory page list includes at least one memory page to be compressed according to a preset algorithm and it is determined that the second type memory page list includes at least one memory page to be compressed, acquiring the at least one memory page to be compressed from the first type memory page list and acquiring the at least one memory page to be compressed from the second type memory page list.

The first type memory page list is a free list, which may be understood as a list that does not store actual memory data, or a free list formed by temporarily not using memory pages storing memory data; the second type of memory page list is an active list, which may be understood as a list that already stores actual memory data, or an active list formed by memory pages that have been recently used and store memory data.

In practical application, at least one memory page to be compressed can be determined from the free list according to a class clock algorithm, a PTE bit6 (DIRTY flag) is defined as PTZIP _ PTE _ YONG, a PTE bit9 (reserved bit) is defined as PTZIP _ PTE _ MIDDLEAGE, YONG mark indicates a hot page which is just accessed to the page, and MIDDLEAGE marks that the page is a cold page; when a page sets the PTZIP _ PTE _ YONG flag, the next scanning changes to the PTZIP _ PTE _ MIDDLEAGE flag, but does not add the compression list; when a page sets PTZIP _ PTE _ MIDDLEAGE flag, the flag is cleared in the next scanning, but not added to the compression list; and only when both marks of one memory page are not available, the memory page is judged to be a cold page and added into the compression list, which represents that the memory page has not been used for some time.

It should be noted that the memory pages to be compressed may be screened from the first type of memory page list, the pages may be screened from the second type of memory page list, and may be screened from the first type of memory page list and the second type of memory page list, which is not limited in this specification.

In the data processing method provided in the embodiment of the present specification, the memory page to be compressed is screened from the memory page list, so that the page descriptor of the memory page is compressed subsequently, the compression of the memory data is realized, and the memory storage space is saved.

Further, the determining that the first type memory page list includes at least one memory page to be compressed according to a preset algorithm includes:

acquiring a page identifier of each memory page in the first memory page list, and determining that the memory page is a to-be-compressed memory page when the page identifier of the memory page is determined to be empty;

correspondingly, the determining, according to a preset algorithm, that the second type memory page list includes at least one memory page to be compressed includes:

and acquiring a page identifier of each memory page in the second memory page list, and determining that the memory page is a to-be-compressed memory page when the page identifier of the memory page is determined to be empty.

Specifically, the compressing unit may obtain a page identifier of each memory page in the first memory page list, determine, when the page identifier of the memory page is determined to be empty, that the memory page is a memory page to be compressed, similarly, obtain a page identifier of each memory page in the second memory page list, and determine, when the page identifier of the memory page is determined to be empty, that the memory page is a memory page to be compressed; in practical application, the cold pages are screened out through a clock-like algorithm, and when the memory page is not used for some time, the memory page is added into a list of memory pages to be compressed, so that page descriptors of the memory pages in the list to be compressed can be compressed subsequently.

In the data processing method provided in the embodiments of the present description, by determining the page identifier of the memory page in the memory page list, the memory page with a page identifier that is empty is screened out as the memory page to be compressed, so as to compress the page descriptor of the memory page to be compressed subsequently, release the idle page descriptor, and reduce the memory management overhead.

Further, screening out a page descriptor with a page identifier being empty, indicating that the page descriptor of the memory page is in an idle state, and compressing the page descriptor in the idle state, so as to reduce the memory of the system; specifically, the determining that the memory page is a memory page to be compressed when it is determined that the page identifier of the memory page is empty includes:

scanning the page identifier of the memory page to obtain an initial page identifier of the memory page;

scanning the page identifier of the memory page again based on a preset time interval to obtain a target page identifier of the memory page;

and under the condition that the initial page identifier and the target page identifier are both determined to be empty, determining the memory page as a memory page to be compressed.

Specifically, the compression unit scans page identifiers in the memory page list to obtain an initial page identifier of the memory page, and scans the page identifiers of the memory page again based on a preset time interval to obtain a target page identifier of the memory page, where it is to be noted that the initial page identifier and the target page identifier are opposite, that is, the first scanned page identifier may be used as the initial page identifier, and the page identifier scanned again may be used as the target page identifier; determining that the memory page is a memory page to be compressed under the condition that the initial page identifier and the target page identifier of the memory page are both empty; in practical application, the screened memory pages to be compressed are cold pages, wherein the page descriptor of each memory page is in an idle state.

In the data processing method provided in the embodiment of the present specification, the page descriptor screened out in the idle state is compressed, so that the memory space is reduced.

Step 206: creating a data management structure for the memory page to be compressed based on the first page descriptor, the second page descriptor, and at least one third page descriptor.

The first page descriptor may be a head page descriptor, the second page descriptor is a first tail page descriptor, and the third page descriptor is a tail page descriptor and is the same as the first tail page descriptor.

The data management structure may be understood as a memory space for storing compressed information, for example, a memory space for storing the first page descriptor and the second page descriptor.

Specifically, a data management structure is established for the screened memory pages to be compressed based on the first page descriptor, the second page descriptor and at least one third page descriptor; it should be noted that, a memory page to be compressed needs 512 page descriptors for management, the first page descriptor is a head page descriptor, and the remaining 511 page descriptors are all tail page descriptors, and when a memory page is looked up and managed by the system, only the head page descriptor needs to be used, and the tail page descriptor is generally in a useless state, and then the contents of the head page descriptor and the tail page descriptor are different, and the contents of the remaining tail page descriptors are the same.

In practical application, in the process of compressing the page descriptors of the memory page to be compressed, the first page descriptor and the second page descriptor need to be stored, so that when the page descriptors of the memory page to be compressed are subsequently decompressed, the page descriptors are filled, the page descriptors of the memory page to be compressed are recovered, and further, a data management structure needs to be created, and the first page descriptor and the second page descriptor are subjected to storage management.

In the data processing method provided in the embodiment of the present specification, a data processing structure is created for a memory page to be compressed to store a first page descriptor and a second page descriptor of the memory page to be compressed, so that after a subsequent system falls into a missing page processing program, a decompression process is completed according to the first page descriptor and the second page descriptor.

Step 208: copying the first page descriptor and the second page descriptor from the memory page to be compressed, storing the copied memory page and the second page descriptor in the data management structure, and deleting the first page descriptor, the second page descriptor and at least one third page descriptor of the memory page to be compressed.

Specifically, the compression unit needs to copy a first page descriptor and a second page descriptor from a memory page to be compressed, store the copied first page descriptor and second page descriptor in a created data management structure, and delete the page descriptors of the memory to be compressed, including the first page descriptor, the second page descriptor, and at least one third page descriptor.

Referring to fig. 3, fig. 3 is a schematic diagram illustrating a data management structure of a data processing method provided by an embodiment of the present specification.

The page table on the left in fig. 3 stores page table entry data, copies the first and second page descriptors from the 512 page descriptors of the memory page, and stores the first and second page descriptors in a data management structure.

After the first page descriptor and the second page descriptor are stored in the data management structure, the address of the data management structure needs to be stored in a page table corresponding to a memory page, so that the address of the data management structure can be conveniently inquired from the page table; specifically, after copying the first page descriptor and the second page descriptor from the memory page to be compressed and storing the copied page descriptors in the data management structure, the method further includes:

and acquiring a structure address of the data management structure, storing the structure address to a page table corresponding to the memory page to be compressed, and establishing a mapping relation between the structure address and the page table corresponding to the memory page to be compressed.

The page table is also called a page table, and is a data structure for translating a virtual address visible to a process into a physical address.

Specifically, the compression unit needs to obtain a structure address of the data management structure, store the structure address to a page table corresponding to a memory page to be compressed, and establish a mapping relationship between the memory page and the data management structure.

In the data processing method provided in the embodiments of the present specification, the address of the data management structure is determined and stored in the page table corresponding to the memory page, so that the page descriptor of the corresponding memory page can be subsequently and quickly determined through the page table entry.

Further, the obtaining the structure address of the data management structure and storing the structure address to the page table corresponding to the memory page to be compressed includes:

determining a first structure address of the data management structure, and storing the first structure address to a first page table entry of a page table corresponding to the memory page to be compressed;

determining a second structure address of the data management structure, and storing the second structure address to a second page table entry of a page table corresponding to the memory page to be compressed, where the structure address includes the first structure address and the second structure address.

Wherein the first fabric address may be understood as a first portion of the data management fabric address; the second structure address may be understood as a second part of the data management structure address, and it should be noted that the first part and the second part together constitute the address of the data management structure.

Specifically, the compression unit determines a first structure address of the data management structure, stores the first structure address to a first page table entry of the page table corresponding to the memory page to be compressed, determines a second structure address of the data management structure, and stores the second structure address to a second page table entry of the page table corresponding to the memory page to be compressed.

In practical application, the data management structure address is a 64-bit address, and the 64-bit address is stored in two page table entries, wherein the first structure address is stored in a first page table entry, and the second structure address is stored in a second page table entry, for example, the 0-31-bit address is stored in the first page table entry, and the 32-63-bit address is stored in the second page table entry, it should be noted that one page table entry may store 64-bit address data, and the 64-bit data management structure address is divided into two page table entries for storage, so that a separate page table entry is not required to store the data management structure, and a part of the page table entries which have already been stored may be selected to continue to store the data management structure address, so that memory space storage is maximized.

In the data processing method provided in the embodiment of the present specification, by storing the address of the data management structure in two page table entries, not only the address of the data management structure can be stored, but also the maximization of the storage memory space is achieved.

After deleting the first page descriptor, the second page descriptor and at least one third page descriptor of the memory page to be compressed, clearing the page table entry identifier of the memory page to be compressed, so as to trap in a missing page processing program and trigger a decompression process when a system kernel needs to access the page description; specifically, after deleting the first page descriptor, the second page descriptor, and at least one third page descriptor of the memory page to be compressed, the method further includes:

and acquiring a page table entry identifier of the memory page to be compressed, and setting the page table entry identifier to be in a first state.

Specifically, after the compression unit compresses the page descriptor of the memory page to be compressed, the page table entry identifier of the memory page to be compressed needs to be set to the first state, in practical application, the page table entry identifier of the memory page to be compressed can be cleared, that is, the page table entry identifier is set to 0, and then when the system kernel needs to access the page descriptor, a page missing processing program will be involved, and a decompression process is triggered, so that the transparent compression effect is achieved.

It should be noted that after the page table entry identifier of the memory page to be compressed is cleared, when the user accesses the memory page, the page descriptor is not found in the page table, the system mechanism is used to trap the process into a missing page processing program, and then the decompression process is automatically triggered, so as to recover the page descriptor of the memory page to be compressed and find corresponding data in the memory page.

In the data processing method provided in the embodiments of the present specification, the page table entry identifier of the memory page to be compressed is cleared, so that a decompression processing program is triggered when a page descriptor is subsequently accessed, and the access to the compressed memory page is realized.

In summary, the data processing method provided in this specification reduces the management overhead by compressing the idle page descriptors in the memory page in the process of compressing the page descriptors of the memory page to be compressed, and can effectively save the memory of the physical server.

Referring to fig. 4, fig. 4 is a flowchart illustrating another data processing method provided in an embodiment of the present specification, which specifically includes the following steps:

step 402: receiving a data processing request aiming at a memory page, and allocating a data storage structure for the memory page based on the data processing request.

The data storage structure may be understood as 8 standard memory pages, and it should be noted that after the 512 page descriptors are recovered, the memory space occupied by the 512 page descriptors is the same as the space occupied by the 8 standard memory pages.

Specifically, after a decompression unit of the compression controller receives a data processing request for a memory page, 8 standard memory pages are allocated to the memory page as a data storage structure, so as to store 512 page descriptors of the memory page to be compressed.

Before the decompression unit receives a data processing request for the memory page, a data query request for data in the memory page needs to be received; specifically, before receiving the data processing request for the memory page, the method further includes:

receiving a data query request, and determining a page table corresponding to a memory page according to a memory page identifier of storage data carried in the data query request;

and under the condition that the page table entry identifier of the memory page is determined to be in the first state, judging whether a corresponding structural address exists in a page table of the memory page, and if so, receiving a data processing request aiming at the memory page.

Specifically, the decompression unit receives the data query request, determines a page table corresponding to the memory page according to a memory page identifier for storing data carried in the data query request, determines whether the page table entry identifier of the memory page is in a first state, determines whether the page table of the memory page has a corresponding structural address, and receives the data processing request for the memory page if the page table entry identifier of the memory page is in the first state.

In practical application, after a memory page to be compressed is compressed, a page table entry identifier is removed, that is, the page table entry identifier is set to a first state, and after a data query request is received, a memory page stored in data to be queried is determined according to a memory page identifier carried in the data query request, so that a page table corresponding to the memory page is determined; under the condition that the page table entry identifier of the memory page in the page table is determined to be in the first state, whether the page table of the memory page has a corresponding structural address is judged, and if yes, a data processing request aiming at the memory page is received.

In the data processing method provided in this specification, for data query by a user, a page table corresponding to a memory page storing data needs to be determined, and if it is determined in the page table that a page table entry identifier of the memory page is in a first state, that is, the page table entry identifier is 0, a structural address corresponding to the memory page is determined in the page table, so that subsequent decompression processing is performed on a page descriptor of the memory page.

Step 404: and obtaining a page table corresponding to the memory page, and determining a structure address of a data management structure corresponding to the memory page based on the page table.

Specifically, the decompression unit obtains a page table corresponding to the memory page, and determines a structure address of a data management structure corresponding to the memory page in the page table, where the data management structure corresponds to the above embodiment, and the data management structure is a storage structure storing a first page descriptor and a second page descriptor of the memory page, which is not described in detail in this embodiment.

Further, the obtaining the page table corresponding to the memory page and determining the structure address of the data management structure corresponding to the memory page based on the page table include:

obtaining a first page table entry and a second page table entry in a page table corresponding to the memory page, and determining a structure address of a data management structure corresponding to the memory page based on the first page table entry and the second page table entry.

Specifically, a first page table entry in a page table corresponding to the memory page stores a part of an address of the data management structure, and a second page table entry stores another part of the address of the data management structure, where the addresses stored in the first page table entry and the second page table entry are merged to obtain a structure address of the data management structure.

In practical applications, the first page table entry and the second page table entry determined in the page table are relative, and it has been explicitly shown in the above embodiment that the 64-bit address of the data management structure is separately stored in different page table entries, and accordingly, in the data lookup process, the address of the data management structure needs to be obtained from different page table entries.

In the data processing method provided in the embodiments of the present specification, the structure address of the data management structure is determined by using different page table entries in the page table, so that the data management structure is subsequently determined from the structure address storing the data management structure, and then the page descriptor corresponding to the memory page is obtained.

Step 406: a first page descriptor and a second page descriptor of the memory page are obtained from the data management structure based on the structure address, and at least one third page descriptor is determined based on the second page descriptor.

Specifically, the decompression unit determines a data management structure corresponding to the memory page according to the structure address, and acquires a first page descriptor and a second page descriptor of the memory page from the data management structure, where, as in the above embodiment, the first page descriptor is a head page descriptor, and the second page descriptor is a tail page descriptor, and then determines at least one third page descriptor based on the second page descriptor, and in practical application, 510 third page descriptors are also determined according to 512 page descriptors, and 511 tail page descriptors other than the first page descriptor are the same.

Step 408: storing the first page descriptor, the second page descriptor, and at least one third page descriptor to the data storage structure.

Specifically, after the third page descriptor of the memory page is determined, the first page descriptor, the second page descriptor and at least one third page descriptor are stored in the data storage structure, and in practical application, 512 restored page descriptors are stored in 8 standard memory pages of the data storage structure, so that decompression of the compressed page descriptors is realized, and the state of the memory page before compression is restored.

After decompressing the compressed memory page, setting the page table entry identifier in the page table entry to 1, and further recovering the state before compression; specifically, after the first page descriptor, the second page descriptor and the at least one third page descriptor are stored in the data storage structure, the method further includes:

and setting the page table entry identifier corresponding to the memory page to be in a second state based on the decompressed first page descriptor, the decompressed second page descriptor and the at least one third page descriptor.

Specifically, for a decompressed memory page, the page table entry identifier corresponding to the memory page is set to the second state, and in practical application, the page table entry identifier is set to 1, and the state before compression is recovered.

In the data processing method provided in the embodiment of the present specification, the page table entry identifier of the memory page is processed, so that whether the page descriptor of the memory page is compressed or not can be subsequently determined according to the state of the memory page identifier, and thus, a transparent page descriptor compression and decompression method is implemented, which can be operated and used economically and efficiently.

To sum up, the data processing method provided in the embodiments of the present specification utilizes the characteristics of the memory page management structure to compress the page descriptor, where the size of the page descriptor is 64B, the compression process does not utilize a compression algorithm that consumes a CPU, but directly releases the memory during compression, and recovers the information in the page descriptor through a copy method during decompression.

Referring to fig. 5, fig. 5 is a flowchart illustrating a processing procedure of a data processing method according to an embodiment of the present specification, which specifically includes the following steps.

Step 502: the user inputs the number of memory pages to be compressed.

Specifically, the preset number of compressed memory pages is input by a user, and in practical application, a system administrator may input the number of memory pages desired to be compressed according to a preset requirement, that is, the preset number of compressed memory pages.

Step 504: and when the memory nursing process determines that the system memory is tense, the decompression unit is triggered, and the compression logic is adjusted.

Specifically, the kernel of the operating system may wake up and check the state of the system memory periodically through a memory care process (kswapd), and when the available memory is lower than the water level, a packer (compression) mechanism may be triggered to work, and measures such as recovering the cache are attempted to release more available memory. ptzip (compression controller) registers a transparent compression processing method with shinker by means of a shrinker mechanism, so that kswapd can trigger a compression unit to work when the system memory is in short supply.

Step 506: the compressing unit determines whether the current compressed memory page reaches the preset number of compressed memory pages, if not, step 508 is executed, and if yes, the process is ended.

Step 508: the compression unit determines a memory page to be compressed.

Specifically, the compression unit starts scanning the memory page list, preferentially scans the idle list, and then scans the active list, because the active large page is more easily reused, the large page which is just compressed triggers the decompression process; then, selecting cold pages by using a clock-like algorithm, and adding the cold pages into a list to be compressed; the clock-like algorithm is implemented as follows: a. defining PTE bit6, DIRTY flag, as PTZIP _ PTE _ YONG, PTE bit9 (reserved bits) as PTZIP _ PTE _ MIDDLEAGE, YONG flag to indicate that the page has just been accessed hot page, MIDDLEAGE flag to indicate that the page is cooler page; b. when a page sets the PTZIP _ PTE _ YONG flag, the next scanning changes to the PTZIP _ PTE _ MIDDLEAGE flag, but does not add the compression list; c. when a page sets PTZIP _ PTE _ MIDDLEAGE flag, the flag is cleared in the next scanning, but not added to the compression list; d. and only when one page has no two marks, the page is not used for a certain time, and if the page is judged to be a cold page, adding the page into a compression list to determine the memory page to be compressed.

Step 510: the compression unit copies a first page descriptor and a second page descriptor of a memory page to be compressed, and places the first page descriptor and the second page descriptor in a data management structure.

Specifically, allocating a ptzip _ hpage (data structure) to store compressed information includes: a descriptor of the head page, a descriptor of the first tail page; then, the 512 page descriptors corresponding to the large page are released, the page descriptors occupy 8 4K pages, the 64-bit address of ptzip _ hpage is stored in the first two PTEs of the memory page, the low-bit address is stored in the high-bit 32 of the first PTE, the high-bit address 32 is stored in the high-bit 32 of the second PTE, and the mapping relation between the memory page and the compression information in the data structure is directly established.

Step 512: the compression unit releases the page descriptors of the memory pages to be compressed.

In practical application, when a page descriptor is compressed, the present flag (page table entry identifier) of the page table entry where the descriptor is located needs to be cleared, so that when the kernel needs to access the page description, a missing page processing program is trapped, and a decompression process is triggered, thereby achieving the effect of transparent compression.

In the data processing method provided in the embodiments of the present description, in the process of compressing the page descriptors of the memory page to be compressed, the idle page descriptors in the memory page are compressed, so that the management overhead is reduced, and the memory of the physical server can be effectively saved.

Referring to fig. 6, fig. 6 is a flowchart illustrating a processing procedure of another data processing method according to an embodiment of the present disclosure, which specifically includes the following steps.

Step 602: a user's data query requests access to the page descriptors of the corresponding memory page.

Step 604: the system kernel access triggers the page fault handler.

Specifically, when any other kernel activity accesses the page descriptor of the compressed memory page, a page missing exception is caused, and a decompression process is triggered; the transparent compression effect of the patent is that only a large-page subsystem senses the compression and decompression processes, and is transparent to other kernel programs using large pages; for example, when a file system uses a large PAGE as a metadata cache, it needs to access a PAGE descriptor, and sets PAGE _ prior flag to trigger the decompression unit to work.

Step 606: the decompression unit determines whether the memory page address causing the page fault is stored in the data management structure of the compressed page descriptor store, if so, step 608 is executed, otherwise, the process ends.

Step 608: the decompression unit allocates a data storage structure to recover the compressed page descriptors.

Specifically, in the page fault exception handling program, 8 memory pages are allocated first, the PTE address where the memory page is located is obtained from the page fault address, and then the address of ptzip _ hpage (data storage structure) is obtained from the PTE.

Step 610: the decompression unit decompresses the page descriptor.

Specifically, the first tail page descriptor information stored in ptzip _ hpage fills the memory page descriptor, sets the present (page table entry identifier) flag in the page table entry corresponding to the decompressed page description, and restores the state before compression.

The data processing method provided by the embodiment of the description utilizes the characteristics of the memory page management structure to compress the page descriptors, the size of the page descriptors is 64B, the compression process does not utilize a compression algorithm which consumes a CPU, the memory is directly released during compression, and the information in the page descriptors is recovered through a copying method during decompression.

Corresponding to the above method embodiment, the present specification further provides a data processing apparatus embodiment, and fig. 7 shows a schematic structural diagram of a data processing apparatus provided in an embodiment of the present specification. As shown in fig. 7, the apparatus includes:

a first receiving module 702, configured to receive a data processing request for a memory page, and obtain and scan a memory page list based on the data processing request;

a first determining module 704, configured to determine at least one memory page to be compressed from the memory page list according to a preset algorithm, where each memory page to be compressed includes a first page descriptor, a second page descriptor, and at least one third page descriptor;

a creating module 706 configured to create a data management structure for the memory page to be compressed based on the first page descriptor, the second page descriptor, and at least one third page descriptor;

a first storage module 708 configured to copy the first page descriptor and the second page descriptor from the memory page to be compressed, store the copied memory page and the second page descriptor in the data management structure, and delete the first page descriptor, the second page descriptor, and at least one third page descriptor of the memory page to be compressed.

Optionally, the apparatus further comprises:

Optionally, the first determining module 704 is further configured to:

Optionally, the apparatus further comprises:

In the data processing apparatus provided in the embodiment of the present specification, in order to compress the page descriptors of the memory page to be compressed, the idle page descriptors in the memory page are compressed, so that the management overhead is reduced, and the memory of the physical server can be effectively saved.

Corresponding to the above method embodiment, the present specification further provides another data processing apparatus embodiment, and fig. 8 shows a schematic structural diagram of another data processing apparatus provided in an embodiment of the present specification. As shown in fig. 8, the apparatus includes:

a second receiving module 802, configured to receive a data processing request for a memory page, and allocate a data storage structure for the memory page based on the data processing request;

a second determining module 804, configured to obtain a page table corresponding to the memory page, and determine, based on the page table, a structure address of a data management structure corresponding to the memory page;

a fetching module 806 configured to fetch a first page descriptor and a second page descriptor of the memory page from the data management structure based on the fabric address, and determine at least one third page descriptor based on the second page descriptor;

a second storing module 808 configured to store the first page descriptor, the second page descriptor, and at least one third page descriptor to the data storage structure.

Optionally, the second determining module 804 is further configured to:

Optionally, the apparatus further comprises:

The data processing apparatus provided in this specification embodiment processes the page table entry identifier of the memory page, so that whether the page descriptor of the memory page is compressed or not can be subsequently determined according to the state of the memory page identifier, and thus a transparent page descriptor compression and decompression method is implemented, which can be economically and efficiently operated and used.

The above is a schematic scheme of two data processing apparatuses of the present embodiment. It should be noted that the technical solution of the data processing apparatus and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the data processing apparatus can be referred to the description of the technical solution of the data processing method.

FIG. 9 illustrates a block diagram of a computing device 900 provided in accordance with one embodiment of the present specification. Components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is coupled to the memory 910 via a bus 930, and a database 950 is used to store data.

Computing device 900 also includes access device 940, access device 940 enabling computing device 900 to communicate via one or more networks 960. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. Access device 940 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 900, as well as other components not shown in FIG. 9, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 9 is for purposes of example only and is not limiting as to the scope of the description. Those skilled in the art may add or replace other components as desired.

Computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), a mobile phone (e.g., smartphone), a wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 900 may also be a mobile or stationary server.

Wherein the processor 920 is configured to execute computer-executable instructions for executing the computer-executable instructions, wherein the steps of the data processing method are implemented when the processor executes the computer-executable instructions.

The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification further provides a computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the data processing method.

The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data processing method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data processing method.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts, but those skilled in the art should understand that the present embodiment is not limited by the described acts, because some steps may be performed in other sequences or simultaneously according to the present embodiment. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for an embodiment of the specification.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the embodiments. The specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method is applied to a first processing unit and comprises the following steps:

2. The data processing method according to claim 1, before receiving the data processing request for the memory page, further comprising:

3. The data processing method according to claim 2, wherein the obtaining of the target number of processed pages for the current memory page when it is received that the memory monitoring system determines that the memory value of the current memory page satisfies the preset memory condition includes:

4. The data processing method according to claim 1 or 2, wherein the memory page list includes a first type memory page list and the second type memory page list;

5. The data processing method according to claim 4, wherein the determining that the first type memory page list includes at least one memory page to be compressed according to a preset algorithm includes:

6. The data processing method according to claim 5, wherein the determining that the memory page is a memory page to be compressed when determining that the page identifier of the memory page is empty includes:

7. The data processing method according to claim 1 or 2, wherein after copying the first page descriptor and the second page descriptor from the memory page to be compressed and storing the copied page descriptor and the second page descriptor in the data management structure, the method further comprises:

8. The data processing method according to claim 7, wherein the obtaining a structure address of the data management structure and storing the structure address to a page table corresponding to the memory page to be compressed includes:

9. A data processing method is applied to a second processing unit and comprises the following steps:

10. The data processing method according to claim 9, wherein the obtaining a page table corresponding to the memory page, and determining a structure address of a data management structure corresponding to the memory page based on the page table includes:

obtaining a first page table entry and a second page table entry in a page table corresponding to the memory page, and determining a structure address of a data management structure corresponding to the memory page based on the first page table entry and the second page table entry;

before receiving the data processing request for the memory page, the method further includes:

11. A data processing apparatus comprising:

12. A data processing apparatus comprising:

13. A computing device, comprising:

a memory and a processor;

the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions to implement the steps of the data processing method of any one of claims 1-8 or 9-10.

14. A computer readable storage medium storing computer instructions which, when executed by a processor, carry out the steps of the data processing method of any one of claims 1 to 8 or 9 to 10.