CN117632782A - Memory expansion and computer persistent memory mechanism and operation system thereof - Google Patents

Abstract

The invention relates to the technical field of computer software, in particular to a memory expansion and computer persistent memory mechanism and an operating system thereof, which are based on mmap and C/C++ programming languages and comprise the following steps: s1, establishing a mmap area and an imaged file; s2, self-making a custom memory distributor; s3, a program switch of the memory distributor; s4, auxiliary macros. Compared with the prior art, the invention can enable the server with 16GB memory to host and run 800GB database. The runtime data and the persistent data residing in the hard disk are integrated, eliminating the need to convert between the runtime data format and the persistent data format.

Description

Memory expansion and computer persistent memory mechanism and operation system thereof

Technical Field

The invention relates to the technical field of computer software, in particular to a memory expansion and computer persistent memory mechanism and a computer readable medium.

Background

The swap space in Linux is used when physical memory (RAM) is full. If the system requires more memory resources and the memory is full, inactive pages in memory will be moved to swap space. The purpose of the computer swap space is simply to relieve system memory space pressure. It is not designed to extend the effective memory capacity.

Linux/POSIX MMAP(2)Memory-Mapped File-I/O:

Linux implements the POSIX standard mmap () system call to correspond an archive to memory. mmap () is a system call that a user process can use to ask the operating system core to image an archive to the memory (i.e., address space) of the process.

This is a functional prototype of mmap ():

void*mmap(void addr,size_t length,int prot,int flags,

int fd,off_t offset)；

addr-this parameter prompts the operating system core to use this address, and the virtual image should start in the virtual memory (i.e., virtual address space) of the process. This value may be designated as NULL to indicate that the core may place the virtual image in any location it deems appropriate. If not NULL, addr should be a multiple of the page size.

length-this parameter specifies the length as the number of bytes of the image. The length should be a multiple of the page size.

prot-protection of mapping memory. The value of prot is the bit or of each of the following units:

PROT_READ-makes the contents of the shadow memory available for the process to READ.

PROT_WRITE-allows the contents of the shadow memory to be written by the process.

PROT_EXEC-allows the contents of the shadow memory to be executed by the program as CPU machine instructions.

flag-various options for controlling images. Some of the more common flag values are described below:

map_anonymous (or map_anon) -allocated ANONYMOUS memories; these pages do not have any file support.

MAP_FILE-preset settings; no designation is required. The image area is supported by regular files.

MAP_fixed-addr is not to be interpreted as a hint: the image is placed exactly at this address, which must be a multiple of the page size.

MAP _ PRIVATE-modifications to the shadow memory region are not visible to other processes that shadow the same file.

Map_shared-modifications to the shadow memory region are visible to other processes that MAP the same file and are ultimately reflected in the file.

fd-an open file descriptor for a file filling a memory area. If MAP_ANONYOUS is specified, fd should be specified as-1.

offset-if this is not an anonymous mapping, the memory map area will be filled with data starting from the offset byte position at the beginning of the file that is opened as file descriptor fd. Should be a multiple of the page size.

Based on the foregoing, it is a matter of urgent need for those skilled in the art how to increase more physical memory and increase the effective memory capacity of a computer system.

Disclosure of Invention

The present invention provides a memory expansion and computer persistent memory mechanism with high practicability, which aims at the defects of the prior art.

The invention further aims to provide a memory expansion and computer persistent memory mechanism operation system which is reasonable in design, safe and applicable.

The technical scheme adopted for solving the technical problems is as follows:

a memory expansion and computer persistent memory mechanism based on mmap and C/C++ programming languages, has the steps of:

s1, establishing a mmap area and an imaged file;

s2, self-making a custom memory distributor;

s3, a program switch of the memory distributor;

s4, auxiliary macros.

Further, in step S1, a configuration file named mmap_mem.cfg is used, which contains mmap archive path, archive size, and image address.

Further, in step S2, the custom memory allocator is used to manage allocation of the mapped memory and release the memory allocator, apply the standard memory allocator to the designated memory area, or customize the custom-designed memory allocator.

Further, in step S3, a memory allocator switch implemented in C/C++, the use_mmap_mem of column 2 is a global variable used as a switch between using the default memory allocator of the system and the custom mmap memory allocator;

the switch is set to each thread switch so that each thread of execution is individually to turn the switch on or off without affecting the other threads being executed.

Further, in step S4, the C/C++ macro MMAP_ MALLOC (statement) is used to turn on the memory allocator switch, execute one or more C/C++ statements, and then turn off the switch.

A memory expansion and computer persistent memory mechanism running system is based on mmap and C/C++ program language, firstly, a mmap area and a mapped file are established, a custom memory distributor is made, a program switch of the memory distributor is configured, and an auxiliary macro is created.

Further, a file archive of the mmap region and the image is established, and a configuration file named mmap_mem.cfg is used, wherein the configuration file comprises a mmap archive path, an archive size and an image address.

Further, the custom memory allocator is used to manage allocation of mapped memory and release memory allocators, apply to designated memory areas for standard memory allocators, or custom-designed memory allocators.

Further, a memory allocator switch implemented in C/C++, with use_mmap_mem of column 2 being a global variable, is used as a switch between using the default memory allocator of the system and the custom mmap memory allocator;

the switch is set to each thread switch so that each thread of execution is individually to turn the switch on or off without affecting the other threads being executed.

Further, the C/C++ macro MMAP_ MALLOC (statement) is used to turn on the memory allocator switch, execute one or more C/C++ statements, and then turn off the switch.

Compared with the computer persistent memory mechanism and the operation system thereof and the prior art, the memory expansion and computer persistent memory mechanism has the following outstanding beneficial effects:

the invention makes it possible to host and run 800GB databases on servers with only 16GB memory. The runtime data and the persistent data residing in the hard disk are integrated. No conversion between runtime data formats and persistent data formats is required.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an operating environment in a memory expansion and computer persistent memory mechanism;

FIG. 2 is a diagram of a prior art computing environment in a memory expansion and computer persistent memory mechanism.

Detailed Description

In order to provide a better understanding of the aspects of the present invention, the present invention will be described in further detail with reference to specific embodiments. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

A preferred embodiment is given below:

a memory expansion and computer persistent memory mechanism in this embodiment is based on mmap and C/C++ programming languages, and has the following steps:

s1, establishing a mmap area and an imaged file;

this step is a standard method of establishing mmap regions. For easier management we use a configuration file named mmap_mem. This configuration file contains mmap file path, file size, and image address.

Example 1:

#mmap_mem.cfg content

path＝/tmp/mmap_mem_file

size＝50G

addr＝0xE0000000000

in the above example, there is a 50GB archive named/tmp/mmap_mem_file. It is mapped to a virtual memory address with hexadecimal value 0xE 0000000000.

Example 2:

fd＝::open(path,O_RDWR|O_CREAT,0666)；assert(fd>＝0)；

off_tfsz＝lseek(fd,0,SEEK_END)；assert(fas＝＝0)；

fsz＝lseek(fd,size-1,SEEK_END)；

size_twsz＝write(fd,“1”,1)；assert(wsz＝＝1)；

mp＝(mmap_mem*)mmap(addr,fsz,PROT_READ|PROT_WRITE|PROT_EXEC,MAP_SHARED|MAP_FIXED,fd,0)；

Madvise(mp,MADV_WILLNEED)；

as shown in example 2, the C language is used to show how files are mapped to memory using Linux mmap (2) system calls. After mapping, the virtual memory address range starts from 0xE0000000000, and the size is 50GB, which is the mapped memory address range.

S2, self-making a custom memory distributor;

memory allocator for managing allocation and release of mapped memory. This may be a standard memory allocator that is used for a given memory area, or a custom-designed memory allocator.

S3, a program switch of the memory distributor;

//thread_local

bool use_mmap_mem＝false；

mmap_mem_mgr*mm＝new mmap_mem_mgr(true)；

mmap_mem*mp＝mm->open_or_create()；

void*operator new(size_t n,const std::nothrow_t&nothrow_value)throw(){

return(mp&&use_mmap_mem)？mp->malloc(n):malloc(n)；}

void*operator new(size_t n){

void*ret＝(mp&&use_mmap_mem)？mp->malloc(n):malloc(n)；

if(！ret)throw std::bad_alloc()；

return ret；}

void operator delete(void*p){

(p&&(mmap_mem::is_mmap_memory(p)))？mp->free(p):free(p)；}

a memory allocator switch implemented in C/c++ is shown. Line 2 use_mmap_mem is a global variable that is used as a switch between using the system default memory allocator and the custom mmap memory allocator.

The switch may also be set to each thread switch. So that each thread of execution can individually turn the switch on or off without affecting the other threads being executed.

S4, auxiliary macros;

#define MMAP_MALLOC(stmt){

use_mmap_mem＝true；

stmt；

use_mmap_mem＝false；}

the C/C++ macro MMAP_ MALLOC (statement) is used to turn on the memory allocator switch, execute one or more C/C++ statements, and then turn off the switch.

The practical use examples are:

setVectorIndex(int cid,int dim,char*col_name,const char*index_type,const char*index_name)

………

Faiss::Index*index；

……..

MMAP_MALLOC(index＝faiss::index_factory(dim,index_type))；

in effect deployed in vector database implementations. The example program code shown in line 2702 shows that the function call of faiss: index_factor () is wrapped by an MMAP_MALLC () macro.

The effect of this is that the vector index created by the faiss:: index_factor () will reside in mmap memory.

As shown in fig. 1, the HDD is a large-capacity hard disk, and is generally 500GB to 1TB or more. The memory is a dynamic memory with a typical capacity of 16GB to 64GB.

Software machine code, which must be loaded into main memory to run;

the runtime data and the persistent data residing in the hard disk are integrated.

The present invention obscures the boundary between runtime data and persistence data. The runtime data and the persistence data are integrated. No conversion between runtime data formats and persistent data formats is required. The runtime data size is no longer limited by the physical memory size.

As shown in fig. 2, the HDD is a large-capacity hard disk, and is typically 500GB to 1TB or more.

The memory is a dynamic memory with a typical capacity of 16GB to 64GB.

Software machine code. The machine code must be loaded into main memory to be able to run.

Runtime data. The runtime data must reside in main memory to be accessed.

Persistent data residing in the hard disk. The persistent data must be read into main memory for use.

Since both programs and data must be loaded into main memory to run, memory size is a limitation of the data size at run time. For example, a database of 800DB cannot be loaded into 16GB memory for operation.

Based on the above method, in a memory expansion and computer persistent memory mechanism operation system in this embodiment, based on mmap and C/c++ program language, first, a mmap area and mapped file are created, a custom memory allocator is made, a program switch of the memory allocator is configured, and an auxiliary macro is created.

Wherein, establish the file of mmap area and image, use the configuration file named mmap_mem.cfg, the said configuration file includes mmap file path, file size and image address.

The custom memory allocator is used to manage allocation of mapped memory and release memory allocators, apply to designated memory areas for standard memory allocators, or custom-designed memory allocators.

Memory allocator switches implemented in C/C++, with use_mmap_mem of column 2 being a global variable, serving as a switch between using the system default memory allocator and the custom mmap memory allocator;

the switch is set to each thread switch so that each thread of execution is individually to turn the switch on or off without affecting the other threads being executed.

The C/C++ macro MMAP_ MALLOC (statement) is used to turn on the memory allocator switch, execute one or more C/C++ statements, and then turn off the switch.

The above-mentioned specific embodiments are merely specific examples of the present invention, and the scope of the present invention is not limited to the specific embodiments, and any suitable changes or substitutions made by those skilled in the art, which conform to the technical solutions described in the claims of the present invention, should fall within the scope of the present invention.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (10)

1. A memory expansion and computer persistent memory mechanism, characterized by mmap and C/c++ based programming language, comprising the steps of:

s1, establishing a mmap area and an imaged file;

s2, self-making a custom memory distributor;

s3, a program switch of the memory distributor;

s4, auxiliary macros.

2. The memory expansion and persistent memory mechanism according to claim 1, wherein in step S1, a configuration file named mmap_mem.cfg is used, said configuration file comprising mmap file path, file size and image address.

3. The memory expansion and computer persistent memory mechanism according to claim 2, wherein in step S2, the custom memory allocator is used to manage allocation of mapped memory and release memory allocators, apply to specified memory areas for standard memory allocators, or custom-designed memory allocators.

4. A memory expansion and computer persistent memory mechanism according to claim 3, wherein in step S3, the memory allocator switch implemented in C/c++, line 2 use_mmap_mem is a global variable used as a switch between using the default memory allocator and the custom mmap memory allocator;

the switch is set to each thread switch so that each thread of execution is individually to turn the switch on or off without affecting the other threads being executed.

5. A memory expansion and computer persistent memory mechanism according to claim 3, wherein in step S4, C/c++ macro mmap_ MALLOC (statement) is used to turn on the memory allocator switch, execute one or more C/c++ statements, and then turn off the switch.

6. A memory expansion and computer persistent memory mechanism operation system is characterized in that a mmap area and a mapped file are firstly established based on mmap and C/C++ programming languages, a custom memory distributor is made, a program switch of the memory distributor is configured, and an auxiliary macro is created.

7. The memory expansion and computer persistent memory mechanism operating system according to claim 6, wherein a mmap region and mapped file is created using a configuration file named mmap_mem.cfg, the configuration file including mmap file path, file size, and map address.

8. The memory expansion and computer persistent memory mechanism operating system according to claim 7, wherein the custom memory allocator is configured to manage allocation of mapped memory and release memory allocators for standard memory allocators for specified memory regions or custom-designed memory allocators.

9. The memory expansion and computer persistent memory mechanism operating system according to claim 8, wherein the memory allocator switch implemented in C/c++, column 2, use_mmap_mem, is a global variable used as a switch between using the default memory allocator and the custom mmap memory allocator;

the switch is set to each thread switch so that each thread of execution is individually to turn the switch on or off without affecting the other threads being executed.

10. The memory expansion and computer persistent memory mechanism operating system according to claim 9, wherein the C/c++ macro mmap_ MALLOC (statement) is configured to turn on the memory allocator switch, execute one or more C/c++ statements, and then turn off the switch.