CN113138859A - General data storage method based on shared memory pool - Google Patents
General data storage method based on shared memory pool Download PDFInfo
- Publication number
- CN113138859A CN113138859A CN202010051558.8A CN202010051558A CN113138859A CN 113138859 A CN113138859 A CN 113138859A CN 202010051558 A CN202010051558 A CN 202010051558A CN 113138859 A CN113138859 A CN 113138859A
- Authority
- CN
- China
- Prior art keywords
- memory
- node
- data
- linked list
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a common data storage method based on a shared memory pool, which comprises the following steps: 1) applying for a shared memory for a target application, and dividing the shared memory into m memory blocks according to different types of contents to be stored by the target application; the head of each memory block stores the manager structure of the memory block, the residual space of the memory block is divided into n memory units, and a shared memory pool is constructed in each memory block; 2) two bidirectional circular linked lists, namely an idle linked list and a busy linked list, are constructed in each memory block; 3) calling a primary storage interface when the target application stores data each time; and performing a plurality of times of memory unit application from the shared memory in the storage interface according to the size of the data to be stored until the data storage is finished, forming the applied memory units into a single chain table, connecting the nodes in the single chain table in series through sub-indexes, and managing the head node of the single chain table by a busy chain table. The invention can reduce the waste of data storage space.
Description
Technical Field
The invention relates to the field of computer memory management and data communication, in particular to a general data storage method based on a shared memory pool.
Background
With the development of scientific and technological progress and computer technology, informatization has been integrated into aspects of modern society, and is closely related to computers from scientific research to industry, and from finance to daily life. Along with the increase of network bandwidth and the popularization of intelligent equipment, computer application scenes are increasingly enriched, real-time interactive services are gradually increased, and various services put higher and higher requirements on the reliability and the processing speed of a server side. Regardless of the single-machine application or cluster deployment, how the server side ensures high availability and low delay has received extensive attention of developers. Therefore, optimizing the performance of server software and fully exploiting the potential of hardware are always important problems to be faced by background developers.
In server programs, data storage and retrieval are important aspects of computer applications, and are a key point for performance optimization. The default application of the system releases the memory mode, and the most common and complex memory use scene is dealt with. Each display call malloc/free corresponds to several underlying system calls (ref: Wujie, Chengxiang, an adaptive variable length block memory pool SVBSMP, computer applications, 2008, Vol. 28). Each memory request usually implies a series of complex algorithms and operations such as exclusive area protection, idle linked list lookup, memory block matching, idle area division or adjacent idle area combination, linked list update and the like. Therefore, frequent allocation and release of memory on the heap leads to performance loss and large amount of memory fragmentation in the system, reducing memory utilization (see: Von Honghua, Xuying, Long distance, Wang, etc., C + + application performance optimization, Beijing: electronics industry Press, 2010). For a specific application, a program constructs a memory pool according to the characteristics of the program, which is the most direct scheme for solving the problem of frequent allocation and memory release of a system. The application program firstly applies for a larger memory and then the program controls the memory to apply for releasing the memory in the memory pool space, which can avoid some extra operations generated by a default method, thereby obtaining higher time efficiency and space efficiency.
The classical memory pool mempool divides a memory into small blocks (units) with the same size, idle blocks form a single linked list, nodes are taken down from the idle chains during each application and hung back to the idle chains during release, and if the space of the current memory area (block) is insufficient, a new memory area (block) is applied and added into the linked list. The method can obtain extremely high time efficiency, the complexity of the memory allocation process is O (1) under most conditions, and the most overhead is the process of initializing and adding the original block chain when a new memory area (block) is opened. This approach is well suited for small space requests.
There are now a large number of web servers running APR _ pool memory management methods from Apache. The APR _ pool divides the multi-level memory pool according to different life cycles of the object and supports the index of the memory block with variable length. The memory management is based on the hierarchy of business processing as a design basis, and pools at different levels have different life cycles. For a single object, application and release pairing does not need to be considered, a batch of objects are destroyed by destroying the pool, the release cost is relatively low, but the space after release is directly returned to the system and cannot be returned to the pool, so that the internal management of the structure also has some disadvantages and is complex in structure.
There are also memory management ideas like adaptive variable length block memory pool SVBSMP (ref: wujie, memorial, an adaptive variable length block memory pool SVBSMP, computer applications, 2008, volume 28). SVBSMP manages two linked lists according to different increasing rates of memory units, manages two memory blocks with different rules of 0-512B and 512B-16KB respectively, and finds out a block meeting the requirement from a target chain and returns the block to a user for each request. And when the idle chain is insufficient, continuously applying for new memory adding management. When the space applied by the user at a time is larger than 512B, the allocation and release work is directly handed to the system for processing, and the recovery space is not included in the SVBSMP memory management. The method is more suitable for the condition that the space of the requested data block has large change and the space requests with different sizes are approximately uniformly distributed. When the space application with a large size concentration is responded, the fragments are linearly increased, and the utilization rate of the memory is reduced.
In the aspect of interprocess communication technology, the Unix-like system provides a plurality of modes for data sharing and synchronization: there are pipes between parent and child processes (pipe), improved half-duplex named pipes (named pipe), signaled signals, chained queues, socket sockets, shared memory, etc. All of them have the characteristics, but in combination, the pipeline, signal and socket modes have the problems of small capacity, low efficiency or easy blockage. In contrast, the shared memory method maps a memory area into a process virtual address space, so that the data sharing can be completed by the relevant process in a user mode, and the shared memory method becomes an interprocess communication method which has the highest efficiency and supports large-capacity data sharing.
Disclosure of Invention
Aiming at the conditions that the heap memory pool is weak in data sharing under the multi-process environment, the traditional memory pool lacks sequential maintenance on stored data, and the variable memory pool is easy to generate external fragments. The invention designs a general data storage method based on a shared memory pool.
The technical scheme of the invention is as follows:
a common data storage method based on a shared memory pool comprises the following steps:
1) applying for a shared memory for a target application, and dividing the shared memory into m memory blocks according to different types of contents to be stored by the target application; the head of each memory block stores the manager structure of the memory block, the residual space of the memory block is divided into n memory units, and a shared memory pool is constructed in each memory block;
2) two bidirectional circular linked lists, namely an idle linked list and a busy linked list, are constructed in each memory block; the idle linked list is used for maintaining idle memory units of the memory blocks, and the busy linked list is used for maintaining busy memory units of the memory blocks; all memory units belong to an idle linked list during initialization, a busy linked list is empty, an idle linked list index iFree of a manager structure points to the head node of the idle linked list of the current memory block, and a busy linked list index iBusy points to the head node index of the busy linked list;
3) calling a primary storage interface when the target application stores data each time; and performing a plurality of times of memory unit application from the shared memory according to the size of data to be stored in the storage interface until the data storage is finished, forming a singly linked list by using the applied memory unit at this time, adding the first memory unit applied at this time as a first node into a busy linked list, using subsequently applied memory units as sub-nodes, hanging the first memory unit at the first node to form the singly linked list, and connecting the nodes in the singly linked list in series through sub-indexes.
Further, the manager structure records the total number totalNum of memory units, the number freeunit, busy unit, busy chain head index, iree, busy chain head index, and memory block number information of the current memory block; each memory unit is provided with a memory unit structure and is used for recording an index value index of the memory unit in a current memory block, a precursor node index pre of the memory unit, a successor node index next of the memory unit, a key value of stored data, a key value length, data content, a data content length, writing time, a subnode number subbcnt, a node subIndex and a subnode successor index value subNextIndex; the node sub-index subIndex refers to an index of a node in a singly linked list.
Further, in step 3), when data is stored each time, the number of memory units required for storing this time is calculated according to the size of the data to be stored and the size of the data area of the memory unit, and if the number of the required memory units exceeds the number of idle units, the application fails; if the number of the applied memory units is less than or equal to the number of the idle units, circularly applying for the idle units according to the required number of the memory units, directly taking the node pointed by the manager structure idle chain table head index iFree during each application, and if the memory interface applies for the idle units for the first time, inserting the taken node in front of the node pointed by the busy chain table head index iBusy as a new head node; through the addition and deletion operation of the bidirectional circulation linked list, the busy linked list added with a new unit and the idle linked list with the iFree node removed are enabled to form the bidirectional circulation linked list again; if the storage interface is not the first idle unit application, the applied memory units are sequentially hung behind the head node of the single-chain table; after applying for the idle unit each time, modifying the number freeNum of the manager structure, subtracting one from the number freeNum, and adding one to the number of busy units, busyNum; and then writing the data key value and the key length into the applied memory unit, writing the data segment into the data area, wherein if the currently applied memory unit is the last memory unit required by the application, the data length is the rest data length, and otherwise, the data length is the length of the data area of the memory unit.
Further, when data are stored, the subnode number subbcnt and the subnode index subIndex of the memory unit structure corresponding to the memory unit are updated, and meanwhile, the index value of the memory unit is recorded into the array to serve as the subNextIndex value of the memory unit applied previously until all the required memory units are stored in the memory unit at the time of application, and the subNextIndex value of the memory unit applied finally is-1.
Further, the number of the memory block is used as the number of the memory pool corresponding to the memory block, each memory pool shares the semaphore number semid, and the number of the memory pool is used as the semaphore array index semnum _ num value of the corresponding memory pool.
Further, when data is searched, the busy linked list is searched according to the input memory block number, the key value and the search sequence, after the first node for storing data is matched, the storage position of the next data segment is found according to the subsequent node information recorded in the first node, and then different segments are connected and returned.
Further, the method for releasing the memory unit through a delete interface comprises the following steps: firstly, positioning a target node corresponding to a memory unit to be released, then finding a first node of data to be deleted, and then deleting singly linked list nodes guided by the first node one by one; if the current deleted node is the first node, taking the first node out of the busy linked list and inserting the first node into the head index iFree of the idle linked list, if the current deleted node is the non-first node, taking the deleted node out of the single linked list and inserting the deleted node into the head index iFree of the idle linked list, and if the head index iFree of the idle linked list is empty, taking the currently deleted node as a new head index iFree node of the idle linked list; when the deleted target node is the busy-link head index iBusy, after the target node is deleted, the new busy-link head index iBusy points to the next node index of the target node.
Furthermore, the method for modifying the data in the memory unit through a modification interface comprises the following steps: firstly, positioning a target memory unit where data to be modified is located, wherein the target memory unit records the total number of memory units occupied by original data; calculating the number of memory units required by the new data according to the new data and the size of the node data area, and if the number of memory units occupied by the original data is equal to the number of memory units required by the new data, directly traversing a singly linked list guided by a target node corresponding to the target memory unit, and rewriting the data area; if the number of occupied memory cells is larger than that of memory cells needed by the new data, rewriting the data area of the memory cells used by the new data, modifying the subnode number subbCNT and the node subIndex subIndex, and deleting the rest memory cells from the busy chain; and traversing a single linked list i guided by a target node corresponding to the target memory cell if the number of the memory cells occupied by the new data is larger than the number of the memory cells occupied by the original data, rewriting the data area, and applying for the new memory cell when the chain end of the single linked list i is reached until the new data storage is completed.
Further, the method for applying for the shared memory for the target application includes: and mapping a segment of physical memory into the address space of the target application through a system call function shmlet.
The invention uses the principle of shared memory (System V standard) and semaphore, and uses the combination of two-way circular chain table group and single chain table as data structure, to realize the shared memory pool for orderly storage and search, and then uses the shared memory pool as basis to package a set of add-delete-modify-check method, so that the user can read and write any type of data in key-value mode. The system provides a storage and retrieval method for the outside in an API form, and specifically comprises the following steps:
1. and (5) dividing the memory. During initialization, a system call function shmget applies for a large enough shared memory for a target application (the shmget system function maps a section of physical memory into a target application address space; Int shmget, size _ t size, Int shmflg), and the same key value is transmitted when different processes call shmget, so that the same memory can be accessed in respective process spaces). The large shared memory is divided into m memory blocks (memory blocks) according to different types of contents to be stored by the application. The head of each memory block takes a fixed-size storage manager (node manager) structure, then the remaining space of the current memory block is divided into n memory units (memory units) with equal size, and a shared memory pool is constructed in each memory block. The manager structure records information such as total number totalNum of memory cells, number of free cells, number of busy cells, free chain head index iree, busy chain head index iBusy, and memory block number of the current memory block, as shown in fig. 1. The memory unit structure records the index value index of the unit in the current memory block, the precursor node index pre, the successor node index next, the key value of the stored data, the key value length, the data content length, the write-in time, the subnode number subCnt, the node subIndex (the index of the node in the single linked list), the subnode successor index value subNextIndex and other information. The specific memory cell structure is shown in fig. 2. Since the shared memory is accessed by multiple processes, the logical address cannot be directly used to mark the memory unit, and since each memory unit is divided into equal size, the offset value of the memory unit in the memory block can be calculated by the index value, so as to obtain the address value. For convenience of description, the memory chains connected in series by the memory cell index values will be referred to as linked lists (two kinds of linked lists are in each memory block, namely a group of circular bidirectional linked lists and a plurality of single linked lists, and a single linked list is formed by one-time memory interface call), and each memory cell in the linked lists is referred to as a node. In order to protect the code of the multi-process access, a mutex semaphore is introduced, and the initial semaphore value is 1, so the administrator records a mutex semaphore number semid and a semaphore group index semnum _ num in addition to the above-mentioned information related to space partition. Because a memory pool is constructed in the memory block space, the serial numbers (0 to m) of the memory blocks can be used as the semaphore group index, thereby achieving the mutually exclusive access of each process to the memory pool.
2. And (4) establishing a linked list. An index number (index value) is set for each memory unit during initialization, the index number represents the position of the memory unit in the memory block, and the index range is from 0 to n-1. The index values of the predecessor and successor units of each memory unit are recorded in each memory unit, so that two bidirectional circular linked lists can be constructed in one memory block to respectively maintain the idle memory unit and the busy memory unit. All memory units belong to an idle linked list during initialization, a busy linked list is empty, an idle linked list index iFree of a manager (node manager) points to a head node of the idle linked list of a current block, a busy linked list index iBusy points to a head node index of a busy linked list (no busy node during initialization, the value is-1), the total number of the memory units of the manager is equal to the number of the idle units, and the number of the busy units is 0. The number of child nodes of the storage unit during initialization defaults to 1(1 means that the child node is self), the node child index value subIndex defaults to 0, and the child node successor index value subIndex is-1 (-1 means no successor child node). The memory management mechanism adopts a mode that one-time interfaces call idle units which are applied for many times and data are stored in segments. The method comprises the following steps that a storage interface is called once every time data are stored, and memory units can be applied for many times in the storage interface according to the size of the data; after the space application is completed, the memory units applied each time are connected through the single linked list, that is, a single linked list (possibly only one node; the nodes in the single linked list are connected in series through the sub-indexes) is generated every time the storage interface is adjusted. The first applied memory unit is used as the first node of the single linked list, the first node is managed by a busy linked list, the later applied memory units are child nodes, the child nodes are hung behind the first node to be queued, and the first node, the child nodes and all the child nodes are serially connected through child node successor indexes. The first node belongs to both the double-linked list and the single-linked list, and the child nodes only belong to the single-linked list guided by the first node.
The chain storage method can store large byte data in segments without requiring continuous memory, so that the scheme can not generate unavailable external fragments, but can form unavailable space at the tail of a memory unit of the last application, or be called memory fragments. Internal fragmentation is temporary and when data is released, the memory cells are returned to be available. In a specific application system, the proper size of the memory unit is set according to the size of data, so that the space waste can be reduced.
3. And (5) calling an interface. Any interface call is firstly connected to the shared memory, the initial address of the target shared memory pool is calculated according to the shared memory address and the memory block number, and then the initial addresses of the manager and the memory unit of the shared memory pool can be obtained. When operating one shared memory pool, executing P operation to check the resource number, when the resource number is greater than or equal to 0, the process is authorized to read and write the shared memory pool, otherwise, the process is called to suspend. When the interface call is finished, executing V operation to increase the signal value, and if a process waits, the waiting process is awakened. This access process is a common operation of the storage interface, retrieval interface, deletion interface, and modification interface mentioned below.
When storing data, calculating how many memory units will be occupied by the storage according to the data size and the data area size of the memory units, and if the number of the needed memory units exceeds the number of the idle units, directly failing to apply the storage; if the number of the applied units is less than or equal to the number of the idle units, the idle units are circularly applied according to the required number, the nodes pointed by the head index iFree of the idle chain of the manager are directly taken in each application, if the idle units are applied by the storage interface for the first time, the taken nodes are inserted before the nodes pointed by the head index iBusy of the busy chain as new head nodes, the busy linked list added with the new units and the idle linked list with the points removed of the iFree nodes form a bidirectional circular linked list again through the addition and deletion operations of the bidirectional circular linked list, at the moment, the nodes newly added into the busy linked list become new iBusy, and the successors of the original iFree serve as new iFree. If the interface does not apply for the idle unit for the first time (namely, the situation of the non-head node), the applied memory units are sequentially hung behind the head node of the single chain table. After each free unit application, the number freeNum of the free units of the manager structure is modified to be reduced by one, and the number busyNum of the busy units is increased by one. And writing the data key value and the key length into the applied memory unit, writing the data segment into the data area, wherein if the application is the required last memory unit, the data length is the residual data length, and if the application is not the last unit, the data length is the length of the data area of the memory unit. And when the data is stored, assigning values to the subnode number subbcnt and the subnode index subIndex of the applied memory unit for the current storage interface, and recording the index value of the memory unit into the array. And circulating the steps until the required unit is applied, recording indexes of a group of memory units applied at this time in the array at the moment, traversing the index array, wherein each element in the array is the subixtindex of the previous node, so that the group of nodes form a singly linked list, the first node in the array is the head node of the singly linked list, and the subixtindex value of the last applied node in the array is-1 (no postcursor node exists in the tail node). Thus, after the storage interface is executed, each memory unit records the key, the value and the length of the data, and also records the position and the number of the associated units.
When data is retrieved, a busy linked list is searched according to the input memory block number, the key value and the search sequence (FIFO or LIFO), after a first node (subIndex value is 0) for storing data is matched, the storage position of the next data fragment can be directly found by means of subsequent node information recorded in the first node, and then different fragments are connected and returned to a user.
Deleting an interface is the process of releasing a memory cell. And deleting the single linked list nodes guided by the first nodes one by one. If the deleted node is the stored first node, the node to be deleted is taken out from the busy linked list and inserted into iFree, if the deleted node is the non-first node, the deleted node is directly taken out from the single linked list and inserted into iFree, if the iFree is empty, the deleted node is used as a new iFree node, and the positions of all nodes in the idle linked list are the same; when the deleted target node is iBusy, after deletion, the new iBusy points to next of the deleted node. When the value of the subtnextindex of a certain node is-1, the node is the last node to be deleted by the deletion interface. The idle chain and the busy chain take the same length, and the node is deleted to form the bidirectional circular linked list again.
The modification interface is the integration of retrieval, deletion and storage. The modified data firstly searches a busy linked list according to the input memory block number, the key value and the search sequence (FIFO or LIFO), and when the key value is matched, the modified data is positioned to a target memory unit. The total number of units occupied by the original data (i.e., the data before modification) is recorded in the target memory unit. Calculating the number of units required by the new data according to the new data (namely the modified data) and the size of the node data area, comparing the number of occupied units of the data before and after modification, and if the number of occupied units is equal to the number of units required by the new data, directly traversing a target node-guided single linked list corresponding to a target memory unit and rewriting the data area; if the number of occupied units is larger than that of units required by the new data, rewriting unit data areas used by the new data, modifying subBnt and subIndex values, and calling a deletion method to delete the excessive units from the busy chain; and if the number of the units occupied by the new data is larger than the existing number of the units of the original data, traversing the target single chain, rewriting the data area of each unit of the original data, applying for a new memory unit when the chain end (subNextIndex is-1), and circulating the process until the new data is stored completely.
The invention has the advantages and beneficial effects that:
the invention adopts a fixed-size separated memory, manages idle and busy memory units by a bidirectional circular chain table group, stores each data request by a single chain table segment, and finally provides an interface function mode for calling an external program. Compared with the traditional memory pool technology, the memory management method has the following advantages:
1. the memory storage system is realized based on a common memory and can be directly used for sharing data among multiple processes.
2. The memory management mechanism adopts a fixed-size separated memory to perform chain storage on the big data application fragmentation, does not require continuous memory units, does not relate to secondary division of a memory block, avoids generating external fragments, and can also consider data requests of different sizes.
3. The data stored in the memory management method is naturally queued according to the time sequence, and can be retrieved, modified and deleted according to a first-in first-out sequence and a last-in first-out sequence. The memory pool constructed based on the method is very suitable to be used as a message input and output buffer area.
4. In the aspect of memory unit application release, each application is to fetch a head node from an idle chain table, the time complexity is O (1), the operation time complexity O (n) is generally retrieved, modified and deleted, and if the operation chain head or the chain tail (as the message buffer condition) is O (1), the time efficiency is high.
Drawings
FIG. 1 is a diagram of a memory pool manager data structure;
FIG. 2 is a diagram of a memory cell data structure;
FIG. 3 is a diagram of a memory management data structure;
FIG. 4 is a diagram of memory pool initialization;
FIG. 5 is a flow chart of a method of storing data 1 and data 2;
FIG. 6 is a flow chart of a method of modifying data 1;
fig. 7 is a flowchart of a method for deleting data 2.
Detailed Description
In order that the above objects, technical solutions and advantages of the present invention will become more apparent, the present invention will be described in detail below with reference to the accompanying drawings in conjunction with specific embodiments.
Hardware environment: phytium FT1500a and Intel (R) Xeon (R) CPU E5-2603.
Operating the system: kylin 4.0.2 and CentOS Linux release 7.6.1810.
1. During initialization, the applied large memory is divided into m memory blocks according to the proportion according to data stored in the shared memory, the head space of each block is used as a manager to store a node _ manager _ t structure, and the space behind the node _ manager _ t is divided into n equal-large memory units. M memory pools are constructed once through a memory partitioning step, wherein block numbers are memory pool numbers (from 0 to m-1), the memory pools share a semaphore number semid, and the number of each memory pool is used as a semaphore array index semnum _ num value. Each memory unit is used as a node, and the nodes are interconnected front and back and are connected end to form a bidirectional circular linked list. The specific data structure is shown in fig. 3. Taking the first block memory pool as an example for explanation, after initialization, the administrator's iFree points to the free chain, the iBusy points to-1 indicates no busy node, semid stores the mutex for multi-process access, and the value of sem _ num is the number of the memory pool. After initialization as in fig. 4.
2. We begin with the storage of two large bytes of data in succession, assuming that the key values for these two data are key1 and key2, respectively, and the data are value1 and value2, respectively. And taking the iFree node from the idle chain every time the memory unit is applied, if the current request stores a data head node, adding the node into a busy chain table to serve as a new head node pointed by the iBusy, and if the current request is a child node, queuing behind the head node. The value1 needs to occupy three memory units, at this time, three nodes with index values of 0, 1 and 2 are taken down from the idle linked list in sequence, the node 0 is the first node of data storage, the nodes 1 and 2 are child nodes, the first node adds a busy linked list as a new head node of the linked list, and since the busy linked list is just created at this time, the predecessor and successor nodes of the node 0 are self-owned, and the child node number subbcnt of the node 0 is 3 (including self-owned). Node 1 and node 2 are added to the singly linked list with node 0 as the head node. When the interface is realized, the process of applying for the idle unit each time is encapsulated into a function, two circular double linked list operations are completed in the function, and the index of the idle unit is used as a function return value. After the required three nodes are applied, the related nodes are connected through the subsequent index (namely the return value iFree of the application function) attribute of the child node to form a data single linked list. Similarly, assuming that value2 occupies two memory cells, two nodes with index numbers 4 and 5 are sequentially taken down when key2 is stored, node 4 is inserted into the busy chain as the first node of data, node 5 is the subsequent child node of node 4, and node 4 and node 5 form a singly linked list. When the key2 storage is complete, the memory effect is as shown in FIG. 5.
3. When data is modified, firstly, a data head node (subIndex value is 0) is matched according to the block number, the key value and the retrieval sequence, a subBnt attribute of the head node is read, and the subBnt records the number of occupied units of the data. If the occupied unit number is equal to the unit number required by the new value, traversing the single linked list guided by the first node, rewriting the data area, and modifying the key value, the key length, the data length, the sub-node number, the sub-index, the sub-node subsequent index and other attribute values of the node; if the occupied units are more than the number of the memory units occupied by the new data, rewriting the memory units needing to be occupied when the new data is stored, and deleting the excessive memory units; and if the number of occupied units is not enough to store new data, applying the new units to continuously store the excessive data after the occupied memory units are rewritten. For example: if the value of the modified key1 is newValue1, newValue1 needs to occupy 4 cells, and then a memory cell is applied after the cell occupied by value1 is rewritten, the newly applied cell index value is 5, and the subsnextindex of the original cell 2 is pointed to the index 5 node to form a new data chain, as shown in fig. 6.
4. The release of the memory corresponds to the deletion of the interface. When the deletion interface is called, firstly, the first node of the data to be deleted is found according to the data retrieval method, then the single linked list nodes guided by the first node are deleted one by one, and the deleted nodes are hung back to the idle linked list and inserted into the iFree nodes. For example: to delete the data key2, the busy chain is first searched to find the first node (subIndex value 0) with key2, which matches the memory location with index 3. The linked list guided by the unit 3 is traversed and the nodes 3 and 4 are deleted in sequence. The post delete key2 post memory area is shown in FIG. 7.
The present invention has been described in detail, but it is apparent that the present invention is not limited to the specific embodiments. It will be apparent to those skilled in the art that various obvious changes can be made therein without departing from the spirit of the process of the invention and the scope of the claims.
Claims (9)
1. A common data storage method based on a shared memory pool comprises the following steps:
1) applying for a shared memory for a target application, and dividing the shared memory into m memory blocks according to different types of contents to be stored by the target application; the head of each memory block stores the manager structure of the memory block, the residual space of the memory block is divided into n memory units, and a shared memory pool is constructed in each memory block;
2) two bidirectional circular linked lists, namely an idle linked list and a busy linked list, are constructed in each memory block; the idle linked list is used for maintaining idle memory units of the memory blocks, and the busy linked list is used for maintaining busy memory units of the memory blocks; all memory units belong to an idle linked list during initialization, a busy linked list is empty, an idle linked list index iFree of a manager structure points to the head node of the idle linked list of the current memory block, and a busy linked list index iBusy points to the head node index of the busy linked list;
3) calling a primary storage interface when the target application stores data each time; and performing a plurality of times of memory unit application from the shared memory according to the size of data to be stored in the storage interface until the data storage is finished, forming a singly linked list by using the applied memory unit at this time, adding the first memory unit applied at this time as a first node into a busy linked list, using subsequently applied memory units as sub-nodes, hanging the first memory unit at the first node to form the singly linked list, and connecting the nodes in the singly linked list in series through sub-indexes.
2. The method as claimed in claim 1, wherein the manager structure records the total number totalNum of memory cells, freeunit number freeNum, busy unit number busyNum, free chain head index iFree, busy chain head index iBusy and memory block number information of the current memory block; each memory unit is provided with a memory unit structure and is used for recording an index value index of the memory unit in a current memory block, a precursor node index pre of the memory unit, a successor node index next of the memory unit, a key value of stored data, a key value length, data content, a data content length, writing time, a subnode number subbcnt, a node subIndex and a subnode successor index value subNextIndex; the node sub-index subIndex refers to an index of a node in a singly linked list.
3. The method according to claim 2, wherein in step 3), each time data is stored, the number of memory units required for the storage is calculated according to the size of the data to be stored and the size of the data area of the memory unit, and if the number of the required memory units exceeds the number of the idle units, the application fails; if the number of the applied memory units is less than or equal to the number of the idle units, circularly applying for the idle units according to the required number of the memory units, directly taking the node pointed by the manager structure idle chain table head index iFree during each application, and if the memory interface applies for the idle units for the first time, inserting the taken node in front of the node pointed by the busy chain table head index iBusy as a new head node; through the addition and deletion operation of the bidirectional circulation linked list, the busy linked list added with a new unit and the idle linked list with the iFree node removed are enabled to form the bidirectional circulation linked list again; if the storage interface is not the first idle unit application, the applied memory units are sequentially hung behind the head node of the single-chain table; after applying for the idle unit each time, modifying the number freeNum of the manager structure, subtracting one from the number freeNum, and adding one to the number of busy units, busyNum; and then writing the data key value and the key length into the applied memory unit, writing the data segment into the data area, wherein if the currently applied memory unit is the last memory unit required by the application, the data length is the rest data length, and otherwise, the data length is the length of the data area of the memory unit.
4. The method as claimed in claim 3, wherein the subnode number subbcnt and the subnode index subIndex of the memory cell structure corresponding to the memory cell are updated when the data is stored, and the index value of the memory cell is recorded in the array until all the required memory cells are requested to be stored this time.
5. The method according to claim 1, wherein the number of a memory block is used as the number of a memory pool corresponding to the memory block, each memory pool shares a semaphore number semid, and the number of the memory pool is used as the semaphore array index semnum _ num value of the corresponding memory pool.
6. The method as claimed in claim 1, wherein when retrieving data, searching the busy chain table according to the input memory block number, key value and search sequence, matching the first node of the stored data, finding the storage position of the next data segment according to the subsequent node information recorded in the first node, and then connecting different segments for returning.
7. The method of claim 1 wherein the memory unit is released through a delete interface by: firstly, positioning a target node corresponding to a memory unit to be released, then finding a first node of data to be deleted, and then deleting singly linked list nodes guided by the first node one by one; if the current deleted node is the first node, taking the first node out of the busy linked list and inserting the first node into the head index iFree of the idle linked list, if the current deleted node is the non-first node, taking the deleted node out of the single linked list and inserting the deleted node into the head index iFree of the idle linked list, and if the head index iFree of the idle linked list is empty, taking the currently deleted node as a new head index iFree node of the idle linked list; when the deleted target node is the busy-link head index iBusy, after the target node is deleted, the new busy-link head index iBusy points to the next node index of the target node.
8. The method of claim 1, wherein the data in the memory unit is modified via a modification interface by: firstly, positioning a target memory unit where data to be modified is located, wherein the target memory unit records the total number of memory units occupied by original data; calculating the number of memory units required by the new data according to the new data and the size of the node data area, and if the number of memory units occupied by the original data is equal to the number of memory units required by the new data, directly traversing a singly linked list guided by a target node corresponding to the target memory unit, and rewriting the data area; if the number of occupied memory cells is larger than that of memory cells needed by the new data, rewriting the data area of the memory cells used by the new data, modifying the subnode number subbCNT and the node subIndex subIndex, and deleting the rest memory cells from the busy chain; and traversing a single linked list i guided by a target node corresponding to the target memory cell if the number of the memory cells occupied by the new data is larger than the number of the memory cells occupied by the original data, rewriting the data area, and applying for the new memory cell when the chain end of the single linked list i is reached until the new data storage is completed.
9. The method of claim 1, wherein the method for applying for the shared memory for the target application comprises: and mapping a segment of physical memory into the address space of the target application through a system call function shmlet.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010051558.8A CN113138859A (en) | 2020-01-17 | 2020-01-17 | General data storage method based on shared memory pool |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010051558.8A CN113138859A (en) | 2020-01-17 | 2020-01-17 | General data storage method based on shared memory pool |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113138859A true CN113138859A (en) | 2021-07-20 |
Family
ID=76808389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010051558.8A Pending CN113138859A (en) | 2020-01-17 | 2020-01-17 | General data storage method based on shared memory pool |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113138859A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113343045A (en) * | 2021-07-29 | 2021-09-03 | 阿里云计算有限公司 | Data caching method and network equipment |
| CN114756591A (en) * | 2022-04-15 | 2022-07-15 | 成都卓讯智安科技有限公司 | Data screening method and system based on bidirectional linked list |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102915276A (en) * | 2012-09-25 | 2013-02-06 | 武汉邮电科学研究院 | Memory control method for embedded systems |
| US20140325177A1 (en) * | 2013-04-24 | 2014-10-30 | International Business Machines Corporation | Heap management using dynamic memory allocation |
| CN106681842A (en) * | 2017-01-18 | 2017-05-17 | 迈普通信技术股份有限公司 | Management method and device for sharing memory in multi-process system |
-
2020
- 2020-01-17 CN CN202010051558.8A patent/CN113138859A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102915276A (en) * | 2012-09-25 | 2013-02-06 | 武汉邮电科学研究院 | Memory control method for embedded systems |
| US20140325177A1 (en) * | 2013-04-24 | 2014-10-30 | International Business Machines Corporation | Heap management using dynamic memory allocation |
| CN106681842A (en) * | 2017-01-18 | 2017-05-17 | 迈普通信技术股份有限公司 | Management method and device for sharing memory in multi-process system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113343045A (en) * | 2021-07-29 | 2021-09-03 | 阿里云计算有限公司 | Data caching method and network equipment |
| CN114756591A (en) * | 2022-04-15 | 2022-07-15 | 成都卓讯智安科技有限公司 | Data screening method and system based on bidirectional linked list |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8261020B2 (en) | Cache enumeration and indexing | |
| US10275489B1 (en) | Binary encoding-based optimizations at datastore accelerators | |
| WO2016187974A1 (en) | Storage space management method and apparatus | |
| US10452655B2 (en) | In-memory cursor duration temp tables | |
| US20120290595A1 (en) | Super-records | |
| EP3688598B1 (en) | Method for reading data stored in a non-volatile cache using rdma | |
| US20210360065A1 (en) | Distributed Metadata Management Method for Distributed File System | |
| CN106570113B (en) | Mass vector slice data cloud storage method and system | |
| CN100424699C (en) | Attribute extensible object file system | |
| US10503693B1 (en) | Method and system for parallel file operation in distributed data storage system with mixed types of storage media | |
| US11977548B2 (en) | Allocating partitions for executing operations of a query | |
| US7505979B2 (en) | Database accelerator | |
| CN113138859A (en) | General data storage method based on shared memory pool | |
| US11061676B2 (en) | Scatter gather using key-value store | |
| CN110162395B (en) | Memory allocation method and device | |
| CN109508241B (en) | Interprocess data exchange method | |
| US11093169B1 (en) | Lockless metadata binary tree access | |
| US11836090B2 (en) | Cache management for search optimization | |
| US11327940B2 (en) | Interlinked tuples in coordination namespace | |
| US11775433B2 (en) | Cache management for search optimization | |
| US11556470B2 (en) | Cache management for search optimization | |
| US11768772B2 (en) | Accumulators corresponding to bins in memory | |
| CN117891796A (en) | HDFS mass small file storage method suitable for multi-read-less-write scene | |
| JP2871755B2 (en) | Split control method in dynamic hash | |
| CN113849548A (en) | Data extraction method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |