CN107786599B - Memory cloud system - Google Patents
Memory cloud system Download PDFInfo
- Publication number
- CN107786599B CN107786599B CN201610741524.5A CN201610741524A CN107786599B CN 107786599 B CN107786599 B CN 107786599B CN 201610741524 A CN201610741524 A CN 201610741524A CN 107786599 B CN107786599 B CN 107786599B
- Authority
- CN
- China
- Prior art keywords
- memory
- storage
- mslice
- data
- cloud server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
Abstract
The invention relates to a memory cloud system, and belongs to the technical field of computer information storage. According to the invention, the client, the memory cloud storage management server and the memory cloud server are interconnected through the Ethernet, a memory cloud system is built by deploying the memory cloud management module, the I/O read-write speed of the storage medium is improved through reasonable memory cloud management design, and the problem of high delay of the storage process is solved. A memory optimization strategy, namely a continuous read-write strategy, is also designed, so that the memory management efficiency is optimized.
Description
Technical Field
The invention relates to the technical field of computer information storage, in particular to a memory cloud system.
Background
In recent years, cloud storage has become a popular subject, and various large internet operators compete to promote cloud storage products and services with different characteristics. Cloud storage has basically adopted a hard disk storage mode in the last decade, that is, a hard disk is used as a storage medium. Since the capacity of the hard disk is increased rapidly (increased by over 1000 times since the middle of 1980 s), the storage mode has a good effect on market application. However, although the capacity of the hard disk is increased rapidly, the performance of the storage method is less than satisfactory, the transmission rate is increased by only 50 times, and the delay is increased by 2 times. The access latency of a hard disk is actually drastically deteriorated if measured in capacity/bandwidth (jimgay's Rule). Meanwhile, the architecture of software is continuously changed along with the continuous development of internet technology, and a common web application architecture generally separates storage from computation, namely, the web application architecture is composed of an application server (a data center which is specially responsible for business logic and front-end presentation of applications) and a storage server. Wherein the application server is stateless, storing only the state of the current browser request. This separate and stateless approach enables the system to scale well to hundreds or thousands of servers, which architecture causes increased latency for data access.
In summary, the hard disk access speed is a major bottleneck in the development of computer systems. In order to solve the problem of data access delay, the industry proposes a new storage idea RAMCloud (memory cloud), and the basic idea is to migrate the main storage center of online data from a hard disk to a DRAM through a memory cluster of a large-scale common server, where the hard disk is only used for backup/archiving. Such a storage scheme may have both large-scale and low-latency I/O features. However, the current research on memory cloud in the industry is relatively small, and a design scheme of a memory cloud system is urgently needed.
Disclosure of Invention
Technical problem to be solved
One of the technical problems to be solved by the invention is as follows: how to improve the I/O read-write speed of the storage medium and solve the problem of high delay of the storage process; the technical problem to be further solved is that: how to optimize memory management efficiency.
(II) technical scheme
In order to solve the technical problem, the invention provides a memory cloud system, which comprises a plurality of clients 1, a memory cloud storage management server 4 and a plurality of memory cloud servers 5; all the three components are accessed to the internet 2, and the memory cloud storage management server 4 and the plurality of memory cloud servers 5 are interconnected through the local ethernet 3;
the memory cloud storage management server 4 is provided with a memory cloud storage management module 7, and is used for receiving a client read-write request of the client 1, establishing an object table, designing a memory allocation mode, and managing memories and data backup and recovery of all the memory cloud servers 5;
the memory cloud server 5 is provided with a memory management module 8 and a data backup management module 9; the memory management module 8 is configured to process a client read-write request of the client 1, complete memory management of the memory cloud server 5, establish a memory table, record use and deletion information of a memory, and complete communication between the memory cloud server 5 and the memory cloud storage management server 4; the data backup management module 9 is configured to complete backup of the memory data to the local disk, constantly keep the local memory data consistent with the disk data, and recover the memory data as needed;
the memory cloud storage client 6 is installed on the client 1 and used for sending a client read-write request and receiving data of the memory cloud storage management server 4 and the memory cloud server 5.
Preferably, the memory cloud server 5 divides an independent storage space for storage, and the independent storage space is composed of two parts: an information area and a data area, the information area for recording information of the data area: block size, bitmap information of the data area; the data area is divided into equal-sized blocks called mslice, and a block size of mslice is recorded in the block size of the information area.
Preferably, the memory cloud storage management module 7 implements memory allocation, memory management and data recovery by maintaining a series of lists, and the table maintained by the memory cloud storage management module 7 includes: the system comprises a memory cloud server management table, an object storage mapping table and a memory cloud server state monitoring table;
the memory cloud server management table summarizes the memory information of all the memory cloud servers 5, the memory cloud server management table takes the memory information of a single memory cloud server 5 as a unit to finally form a summary table, and each unit consists of a memory cloud server ID, a synchronous identifier and an information area thereof; the memory allocation condition of the memory cloud server 5 is displayed from the summary table, and when the memory use condition of the memory cloud server 5 changes, the memory cloud storage management module 7 synchronously records the change to a memory cloud server management table;
the object storage mapping table records the client 1 object and a storage area corresponding to the client 1 object;
the memory cloud server state monitoring table records the number of the memory cloud servers 5, the current connection state and the data backup state of all the memory cloud servers 5;
preferably, the memory cloud storage management module 7 is configured to execute the following processing procedures:
(1) initialization: scanning all the memory cloud servers 5, acquiring memory information area information from each memory cloud server 5, and establishing a memory cloud server management table;
(2) establishing a memory cloud server state monitoring table;
(3) regularly maintaining the memory cloud server state monitoring table, and updating the memory cloud server management table if the memory cloud server state monitoring table changes;
(4) waiting for a request of the client 1; when a reading request of the client 1 is received, retrieving the ID of the storage object, searching the corresponding storage space according to the ID of the storage object, if the corresponding ID of the storage object is not found, returning a not-found message, otherwise, sending the storage space distribution information of the storage object to the client 1; when a write request of a client 1 is received, distributing an ID (identity) to a storage object, and acquiring the size of a required storage space;
(5) reading the memory cloud server management table, carrying out consistency check, and if the memory cloud server management table is not consistent, carrying out synchronization;
(6) allocating space according to a memory cloud server management table, marking the allocated bitmap as pre-allocation, establishing an object storage mapping table, and respectively sending the object storage mapping table to the client 1 and the memory cloud server 5;
(7) waiting for the memory cloud server 5 to send an end message, setting the corresponding bitmap mark as allocation, and checking the backup state;
when a client 1 requests to delete an object, firstly, a storage object ID is sent to a related memory cloud server 5, when a deletion success message sent by the memory cloud server 5 is received, an identifier of a bitmap corresponding to a memory cloud server management table is cleared, a record corresponding to an object storage mapping table is deleted, a result is fed back to the client 1, and if the deletion failure message is received, the deletion failure message is fed back to the client 1.
Preferably, the memory management module 8 includes a memory allocation recovery module, a memory optimization module and a data read-write module;
the memory allocation and recovery module is used for allocating and recovering the memory. The memory allocation refers to an object storage mapping table sent by a memory cloud storage management server 4 for processing, when data needs to be stored, the data is written into a corresponding mslice according to the object storage mapping table of the belonging object, a corresponding bitmap is identified, and after the data is written into a memory area, the corresponding bitmap in the object storage mapping table needs to be identified as allocated; when the memory is recycled, the mslice used by the storage object is searched according to the ID of the storage object, the content is deleted, and the bitmap identification is emptied; feeding back the result to the memory cloud storage management server 4;
the memory optimization module is used for optimizing the whole storage process by combining fragmented storage spaces and creating continuous idle areas, the memory optimization is only carried out in the idle state, an original object storage mapping table is backed up before the memory optimization, the bitmap identification of the information area and the object storage mapping table are updated after the memory optimization, and the tables on the memory cloud storage management server 4 are synchronized;
the data read-write module is used for receiving and sending data of the client 1, and when the memory cloud server 5 receives the data from the client 1, processing is carried out according to the object storage mapping table.
Preferably, the data backup management module 9 is configured to backup local memory data to a local disk, first select a storage area larger than the memory storage area, then perform first synchronization, when the client 1 stores or deletes data, perform data reading and writing of the memory area by the memory management module 8, and perform data reading and writing of the disk area by the data backup management module 9; and after the memory cloud server 5 performs memory optimization, synchronizing the disk backup data.
Preferably, the memory optimization module is specifically configured to execute the following procedures:
(1) searching for an idle mslice, if not, ending, otherwise, turning to the step (2);
(2) finding an idle mslice, marking the idle mslice as A, numbering the idle mslice as p1, if the idle mslice is the last mslice, ending the process, and otherwise, turning to the step (3);
(3) if A is the first mslice or the mslice immediately preceding to the first mslice is idle mslice, continuing to look for the next idle mslice backwards, returning to the step (1), otherwise, recording the number of A, acquiring an object X to which the mslice immediately preceding to the first mslice belongs, and turning to the step (4);
(4) with p1 as a starting point, searching mslices belonging to an object X one by one backwards, if not, continuing to search a next idle mslice backwards, returning to the step (1), otherwise, finding the mslice belonging to the object X, marking as B, numbering as p2, and exchanging storage positions of the A and the B;
(5) and (4) taking p2 as a starting point, searching for the next idle mslice backwards and returning to the step (1).
(III) advantageous effects
According to the invention, the client, the memory cloud storage management server and the memory cloud server are interconnected through the Ethernet, a memory cloud system is built by deploying the memory cloud management module, the I/O read-write speed of the storage medium is improved through reasonable memory cloud management design, and the problem of high delay of the storage process is solved. A memory optimization strategy, namely a continuous read-write strategy, is also designed, so that the memory management efficiency is optimized.
Drawings
Fig. 1 is a general architecture diagram of a memory cloud system according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an internal architecture of a memory cloud system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data storage area in the system according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating the system memory optimization results according to an embodiment of the present invention.
Wherein:
1 client 2 internet
3 Ethernet 4 memory cloud storage management server
5 memory cloud server 6 memory cloud storage client
7 memory cloud storage management module and 8 memory management module
9 data backup management Module 10 data flow
11 control flow 12 data store
13 mslice 14 idle mslice of different objects
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
As shown in fig. 1, an embodiment of the present invention provides a memory cloud system, which uses a DRAM to replace a conventional disk, and is composed of a plurality of clients 1, a memory cloud storage management server 4, and a plurality of memory cloud servers 5, all of which are connected to the internet 2, and at the same time, the memory cloud storage management server 4 and the plurality of memory cloud servers 5 are interconnected via a local ethernet 3.
To achieve fast, low latency access for the user, the architecture is shown in FIG. 2, and includes: the system comprises a memory cloud storage management module 7 installed on the memory cloud storage management server 4, a memory management module 8 and a data backup management module 9 installed on the memory cloud server 5, and a memory cloud storage client 6.
The memory cloud storage management module 7 is used for receiving a client read-write request of the client 1, establishing an object table, designing a memory allocation mode, and managing memories, data backup and recovery of all the memory cloud servers 5;
the memory management module 8 is configured to process a client read-write request of the client 1, and complete memory management of the memory cloud server 5: distributing, recycling and optimizing, establishing a memory table, recording the use and deletion information of the memory, and completing the communication between the memory cloud server 5 and the memory cloud storage management server 4; the data backup management module 9 is configured to complete backup of the memory data to the local disk, constantly keep the local memory data consistent with the disk data, and recover the memory data as needed;
the memory cloud storage client 6 is used for sending a client read-write request and receiving data of the memory cloud storage management server 4 and the memory cloud server 5.
For the purpose of aspect management, the memory cloud server 5 will divide a large independent memory space for storage, and the space is composed of two parts: an information area and a data area. The information area is used for recording some specific information of the data area: block size, bitmap information of the data area, other reserved information. For the data area, the system divides the data area into equal-sized blocks called mslice, the block size of which is recorded in the block size of the information area.
1. Design of memory cloud storage management module 7
The memory cloud storage management module 7 realizes functions of memory allocation, memory management, data recovery and the like through maintenance of a series of lists. The table maintained by the memory cloud storage management module 7 includes: the system comprises a memory cloud server management table, an object storage mapping table and a memory cloud server state monitoring table.
The memory cloud server management table summarizes the memory information of all the memory cloud servers 5, the memory cloud server management table takes the memory information of a single memory cloud server 5 as a unit to finally form a summary table, and each unit consists of a memory cloud server ID, a synchronous identifier and an information area thereof. The memory allocation condition of a certain memory cloud server 5 can be directly obtained from the summary table, and when the memory usage condition of a certain memory cloud server 5 changes, the change needs to be synchronously recorded to the memory cloud server management table. By establishing the memory cloud server management table, when there is a write request of the cloud storage client 1, the memory cloud server 5 does not need to be accessed one by one and the memory cloud server is directly pre-allocated locally.
The object of the system is defined as a storage unit of the client 1, and can be a file, a partition, a disk and the like. The object memory mapping table records the client 1 object and the corresponding memory area.
The memory cloud server state monitoring table records the number of the memory cloud servers 5, the current connection state and the data backup state of all the memory cloud servers 5, and the like.
The basic processing flow of the memory cloud storage management module 7 is as follows:
(1) initializing, scanning all the memory cloud servers 5, acquiring memory information area information of the server from each memory cloud server 5, and establishing a memory cloud server management table;
(2) establishing a memory cloud server state monitoring table;
(3) regularly maintaining the memory cloud server state monitoring table, and updating the memory cloud server management table if the memory cloud server state monitoring table changes;
(4) waiting for a request of the client 1;
(5) when a reading request of the client 1 is received, retrieving the ID of the storage object, searching the corresponding storage space according to the ID of the storage object, if the corresponding ID of the storage object is not found, returning a not-found message, otherwise, sending the storage space distribution information of the storage object to the client 1;
(6) when a write request of a client 1 exists, distributing an ID (identity) to a storage object, and acquiring the size of a required storage space;
(7) reading the memory cloud server management table, carrying out consistency check, and if the memory cloud server management table is not consistent, carrying out synchronization;
(8) allocating space according to a memory cloud server management table, marking the allocated bitmap as pre-allocation, establishing an object storage mapping table, and respectively sending the object storage mapping table to a client 1 and a memory cloud server 5;
(9) and waiting for the memory cloud server 5 to send an end message, and marking the corresponding bitmap as allocation. Checking the backup state;
when the client 1 requests to delete the object, the storage object ID is firstly sent to the relevant memory cloud server 5, when a deletion success message sent by the memory cloud server 5 is received, the identification of the bitmap corresponding to the memory cloud server management table is cleared, the record corresponding to the object storage mapping table is deleted, and the result is fed back to the client 1. And if the message of the deletion failure is received, directly feeding back the message of the deletion failure to the client 1.
The memory cloud storage management module 7 also has data recovery functions including local data recovery and memory cloud server data recovery.
And local data recovery is realized by reconstructing a memory cloud server management table, an object storage mapping table and a memory cloud server state monitoring table through data on the connected memory cloud server 5.
The data recovery process of the memory cloud server 5 comprises the following steps: when the memory data of a certain memory cloud server 5 is lost or damaged, the backup data is restored to the memory from the local disk.
2. Design of memory management module 8
The memory management module 8 is composed of a memory allocation recovery module, a memory optimization module and a data read-write module.
The memory allocation and recovery module completes allocation and recovery of the memory. The memory allocation refers to the object storage mapping table sent by the memory cloud storage management server 4 for processing. When data needs to be stored, writing the data into the corresponding mslice according to the object storage mapping table of the object to which the data belongs, and identifying the corresponding bitmap. After the data is written into the memory area, the corresponding bitmap in the object storage mapping table needs to be identified as allocated. And when the memory is recycled, searching the mslice used by the storage object according to the ID of the storage object, deleting the content and emptying the bitmap identification. And feeding back the result to the memory cloud storage management server 4.
The memory optimization module optimizes the whole storage process by combining fragmented storage space and creating continuous free areas. If a common read-write process is used, the influence on the performance is not great if the memory is optimized, but the system adopts a continuous read-write strategy, so the system performance can be improved. The continuous read-write strategy means that when the read-write mslices are continuous, a group of continuous mslices are called msgments, and the system calculates the starting and stopping positions of the msgments so that multiple read-write operations are completed through one read-write operation. In order not to affect the read-write performance of the system, the memory optimization can be carried out only in idle time.
Memory optimization is a very complex process, and the efficiency of optimization algorithms varies. In order to find a suitable algorithm, the characteristics of the data storage of the system are firstly analyzed, fig. 3 is a simple example of the characteristics of the data storage of the system, a data storage area 12 comprises an mslice 13 and an idle mslice 14 for storing three different objects, which can be easily obtained from fig. 3, and the data stored by the storage system has the following characteristics:
(1) the data of the same object are stored in sequence;
(2) free areas exist only between msegments of different objects.
The aim of the algorithm is to combine the idle mslices and simultaneously ensure the centralized storage of the same object as much as possible, thereby improving the data reading and writing efficiency. To achieve the above objective, the algorithm of the present system is as follows:
(1) searching for an idle mslice, if not found, ending, otherwise, switching to (2);
(2) finding an idle mslice, marking the idle mslice as A, numbering the idle mslice as p1, if the idle mslice is the last mslice, ending the process, and if the idle mslice is not the last mslice, switching to the step (3);
(3) if A is the first mslice or the mslice adjacent to the first mslice in the front direction is idle mslice, continuously searching the next idle mslice backwards, turning to step (1), otherwise, recording the number of A and acquiring an object X to which the mslice adjacent to the first mslice in the front direction belongs;
(4) with p1 as a starting point, searching mslices belonging to an object X one by one backwards, if not, continuing to search a next idle mslice backwards, turning to step (1), otherwise, finding the mslices belonging to the object X, recording B, wherein the number of the mslices is p2, and A and B exchange storage positions;
(5) starting from p2, look backward for the next free mslice, and go to (1).
The steps realize the continuous storage of the same object mslice and the idle mslice through a series of exchange mslices.
Taking fig. 3 as an example, according to the above algorithm, the data exchange process is (where mslice [ i ] represents mslice with number i before the algorithm starts):
(1) finding the current 1 st block idle mslice: mslice [4], remember immediately forward of mslice: mslice [3], where the object is A, so it needs to find the next mslice of object A: mslice [8], post-permutation mslice [4] and mslice [8 ];
(2) continue to look down for free mslcie: mslice [5], immediately forward of mslice is mslice [8], so mslice [20] are found backward, and mslice [5] and mslice [20] are replaced;
(3) repeating step (1) to find mslice [4], at which time, by permutation, mslice [4] is located at position number 8 to find the immediately preceding mslice: mslice [7], the object to which the mslice belongs is B, the mslice belonging to the object B is searched backwards: mslice [9], replacement mslice [4] with mslice [9 ];
(4) according to the algorithm, mslice [4] and mslice [10], mslice [4] and mslice [11], mslice [4] and mslcie [18] are sequentially replaced, and after replacement, mslice [4] is located at a position with the number being 18;
(5) continue to look for the next free mslice: mslice [12], immediately preceding mslice being mslice [11], belonging to object B, and searching backward for mslice belonging to object B: mslice [19], substituted mslice [12] and mslice [19 ];
(6) continue to look for the next free mslice: mslice [17], immediately preceding mslice as mslice [16], belonging to object C, and searching backward for mslice of belonging object C: mslice [22], replacing mslice [17] and mslice [22], according to the flow, no mslice needing replacing exists at this time, the flow is ended, and the memory area after replacing is shown in FIG. 4.
Through the steps, all blank mslices are combined, and the data of the same object basically realizes centralized storage.
Before optimizing the memory, the original object storage mapping table needs to be backed up. After the memory is optimized, the information area bitmap identification and the object storage mapping table need to be updated, and the table on the memory cloud storage management server is synchronized in time.
The data read-write module is responsible for receiving and sending client data. When the memory cloud server 5 receives data from the client 1, processing is performed according to the object memory mapping table.
4. Design of data backup management module 9
The data backup management module 9 is responsible for backing up the local memory data to the local disk. The system firstly selects a storage area larger than the memory storage area, and then carries out the first synchronization, wherein the process is mainly to synchronize the data of the memory area information area. When the client 1 stores or deletes data, the memory management module 8 is responsible for reading and writing data of the memory area, and the data backup management module 9 is responsible for reading and writing data of the disk area. The data backup process is similar to the data read-write process of the memory management module 8, but the performance is relatively low. After the memory cloud server 5 performs memory optimization, disk backup data needs to be synchronized in time, and the specific method is to compare the updated object storage mapping table with the original object storage mapping table and adjust the data to the recording sequence of the updated object storage mapping table.
5. Design of memory cloud storage client 6
The memory cloud storage client 6 provides a visual operation interface for a user, and simultaneously issues the use requirements of the customer.
Through the mutual cooperation of the modules, the functional requirements of the memory cloud system are basically met.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (1)
1. The memory cloud system is characterized by comprising a plurality of clients (1), a memory cloud storage management server (4) and a plurality of memory cloud servers (5); all the three components are accessed to the Internet (2), and the memory cloud storage management server (4) and the plurality of memory cloud servers (5) are interconnected through the local Ethernet (3);
the memory cloud storage management server (4) is provided with a memory cloud storage management module (7) which is used for receiving a client read-write request of the client (1), establishing an object table, designing a memory distribution mode and managing memories, data backup and recovery of all the memory cloud servers (5);
the memory cloud server (5) is provided with a memory management module (8) and a data backup management module (9); the memory management module (8) is used for processing a client read-write request of the client (1), completing memory management of the memory cloud server (5), establishing a memory table, recording use and deletion information of the memory, and completing communication between the memory cloud server (5) and the memory cloud storage management server (4); the data backup management module (9) is used for completing the backup of the memory data to the local disk, constantly keeping the consistency of the local memory data and the disk data, and recovering the memory data as required;
the memory cloud storage client (6) is installed on the client (1) and used for sending a client read-write request and receiving data of the memory cloud storage management server (4) and the memory cloud server (5);
the memory cloud server (5) divides an independent storage space for storage, and the independent storage space consists of two parts: an information area and a data area, the information area for recording information of the data area: block size, bitmap information of the data area; dividing the data area into blocks with equal size, called mslice, wherein the block size of the mslice is recorded in the block size of the information area;
the memory cloud storage management module (7) realizes memory allocation, memory management and data recovery through maintaining a series of lists, and the table maintained by the memory cloud storage management module (7) comprises: the system comprises a memory cloud server management table, an object storage mapping table and a memory cloud server state monitoring table;
the memory cloud server management table summarizes the memory information of all the memory cloud servers (5), the memory cloud server management table takes the memory information of a single memory cloud server (5) as a unit to finally form a summary table, and each unit consists of a memory cloud server ID, a synchronous identifier and an information area thereof; displaying the memory allocation condition of the memory cloud server (5) from the summary table, and when the memory use condition of the memory cloud server (5) changes, synchronously recording the change to a memory cloud server management table by a memory cloud storage management module (7);
the object storage mapping table records the object of the client (1) and a storage area corresponding to the object;
the memory cloud server state monitoring table records the number of the memory cloud servers (5), the current connection state and the data backup state of all the memory cloud servers (5);
the memory cloud storage management module (7) is used for executing the following processing procedures:
1) initialization: scanning all the memory cloud servers (5), acquiring memory information area information from each memory cloud server (5), and establishing a memory cloud server management table;
2) establishing a memory cloud server state monitoring table;
3) regularly maintaining the memory cloud server state monitoring table, and updating the memory cloud server management table if the memory cloud server state monitoring table changes;
4) waiting for a request of the client (1); when a reading request of the client (1) is received, retrieving the ID of the storage object, searching the corresponding storage space according to the ID of the storage object, if the corresponding ID of the storage object is not found, returning a not-found message, otherwise, sending the storage space distribution information of the storage object to the client (1); when a write request of a client (1) is received, distributing an ID (identity) to a storage object, and acquiring the size of a required storage space;
5) reading the memory cloud server management table, carrying out consistency check, and if the memory cloud server management table is not consistent, carrying out synchronization;
6) allocating space according to a memory cloud server management table, marking the allocated bitmap as pre-allocation, establishing an object storage mapping table, and respectively sending the object storage mapping table to a client (1) and a memory cloud server (5);
7) waiting for the memory cloud server (5) to send an end message, and checking the backup state by taking the corresponding bitmap mark as allocation;
when a client (1) requests to delete an object, firstly, a storage object ID is sent to a related memory cloud server (5), when a deletion success message sent by the memory cloud server (5) is received, an identifier of a bitmap corresponding to a memory cloud server management table is cleared, a record corresponding to an object storage mapping table is deleted, a result is fed back to the client (1), and if the deletion failure message is received, the deletion failure message is fed back to the client (1);
the memory management module (8) comprises a memory allocation recovery module, a memory optimization module and a data read-write module;
the memory allocation and recovery module is used for allocating and recovering the memory; the memory allocation refers to an object storage mapping table sent by a memory cloud storage management server (4) for processing, when data needs to be stored, the data is written into a corresponding mslice according to the object storage mapping table of the belonging object, a corresponding bitmap is identified, and after the data is written into a memory area, the corresponding bitmap in the object storage mapping table needs to be identified as allocated; when the memory is recycled, the mslice used by the storage object is searched according to the ID of the storage object, the content is deleted, and the bitmap identification is emptied; feeding back the result to a memory cloud storage management server (4);
the memory optimization module is used for optimizing the whole storage process by combining fragmented storage spaces and creating continuous idle areas, the memory optimization is only carried out in the idle state, an original object storage mapping table is backed up before the memory optimization, the bitmap identification of the information area and the object storage mapping table are updated after the memory optimization, and the table on the memory cloud storage management server (4) is synchronized;
the data read-write module is used for receiving and sending data of the client (1), and when the memory cloud server (5) receives the data from the client (1), processing is carried out according to the object storage mapping table;
the data backup management module (9) is used for backing up local memory data to a local disk, firstly, a storage area larger than a memory storage area is selected, then, first synchronization is carried out, when the client (1) stores or deletes the data, the memory management module (8) carries out data reading and writing of the memory area, and the data backup management module (9) carries out data reading and writing of a disk area; after the memory cloud server (5) performs memory optimization, synchronizing disk backup data;
the memory optimization module is specifically configured to execute the following procedures:
(1) searching for an idle mslice, if not, ending, otherwise, turning to the step (2);
(2) finding an idle mslice, marking the idle mslice as A, numbering the idle mslice as p1, if the idle mslice is the last mslice, ending the process, and otherwise, turning to the step (3);
(3) if A is the first mslice or the mslice immediately preceding to the first mslice is idle mslice, continuing to look for the next idle mslice backwards, returning to the step (1), otherwise, recording the number of A, acquiring an object X to which the mslice immediately preceding to the first mslice belongs, and turning to the step (4);
(4) with p1 as a starting point, searching mslices belonging to an object X one by one backwards, if not, continuing to search a next idle mslice backwards, returning to the step (1), otherwise, finding the mslice belonging to the object X, marking as B, numbering as p2, and exchanging storage positions of the A and the B;
(5) and (4) taking p2 as a starting point, searching for the next idle mslice backwards and returning to the step (1).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610741524.5A CN107786599B (en) | 2016-08-26 | 2016-08-26 | Memory cloud system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610741524.5A CN107786599B (en) | 2016-08-26 | 2016-08-26 | Memory cloud system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107786599A CN107786599A (en) | 2018-03-09 |
| CN107786599B true CN107786599B (en) | 2020-07-17 |
Family
ID=61441021
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610741524.5A Active CN107786599B (en) | 2016-08-26 | 2016-08-26 | Memory cloud system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107786599B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111176549B (en) * | 2018-11-13 | 2021-05-18 | 杭州海康威视系统技术有限公司 | Data storage method and device based on cloud storage and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1773468A (en) * | 2004-11-12 | 2006-05-17 | 中国科学院计算技术研究所 | A network internal storage service system and its constituting method |
| CN101013400A (en) * | 2007-01-30 | 2007-08-08 | 金蝶软件（中国）有限公司 | Method and apparatus for cache data in memory |
| CN101950273A (en) * | 2010-09-26 | 2011-01-19 | 中国人民解放军海军工程大学 | Dynamic management method of DSP data area |
| CN102866953A (en) * | 2011-07-08 | 2013-01-09 | 风网科技（北京）有限公司 | Storage management system and storage management method thereof |
| CN104102460A (en) * | 2014-07-23 | 2014-10-15 | 浪潮(北京)电子信息产业有限公司 | Cloud computing-based memory management method and device |
| CN105404590A (en) * | 2015-11-10 | 2016-03-16 | 浪潮(北京)电子信息产业有限公司 | Memory block combination method and system |
| CN106095577A (en) * | 2016-06-08 | 2016-11-09 | 东北林业大学 | The optimized treatment method of shared drives based on many path processors and device |
-
2016
- 2016-08-26 CN CN201610741524.5A patent/CN107786599B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1773468A (en) * | 2004-11-12 | 2006-05-17 | 中国科学院计算技术研究所 | A network internal storage service system and its constituting method |
| CN101013400A (en) * | 2007-01-30 | 2007-08-08 | 金蝶软件（中国）有限公司 | Method and apparatus for cache data in memory |
| CN101950273A (en) * | 2010-09-26 | 2011-01-19 | 中国人民解放军海军工程大学 | Dynamic management method of DSP data area |
| CN102866953A (en) * | 2011-07-08 | 2013-01-09 | 风网科技（北京）有限公司 | Storage management system and storage management method thereof |
| CN104102460A (en) * | 2014-07-23 | 2014-10-15 | 浪潮(北京)电子信息产业有限公司 | Cloud computing-based memory management method and device |
| CN105404590A (en) * | 2015-11-10 | 2016-03-16 | 浪潮(北京)电子信息产业有限公司 | Memory block combination method and system |
| CN106095577A (en) * | 2016-06-08 | 2016-11-09 | 东北林业大学 | The optimized treatment method of shared drives based on many path processors and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107786599A (en) | 2018-03-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100399327C (en) | Managing file system versions | |
| CN102346695B (en) | Scalable segment-based data de-duplication system and method for incremental backups | |
| US7774565B2 (en) | Methods and apparatus for point in time data access and recovery | |
| JP2020038623A (en) | Method, device, and system for storing data | |
| CN103116661B (en) | A kind of data processing method of database | |
| CN105938457B (en) | Filter method, device and the data reading system of data | |
| CN101777017B (en) | Rapid recovery method of continuous data protection system | |
| CN107357680A (en) | Data processing equipment and the method for data processing | |
| CN105574217B (en) | The method of data synchronization and device of distributed relation database | |
| CN100424699C (en) | Attribute extensible object file system | |
| CN104391873A (en) | Database operation separation method and database operation separation system | |
| CN103544045A (en) | HDFS-based virtual machine image storage system and construction method thereof | |
| CN110058822A (en) | A kind of disk array transverse direction expanding method | |
| CN103458023A (en) | Distribution type flash memory storage system | |
| CN102662992A (en) | Method and device for storing and accessing massive small files | |
| CN102541985A (en) | Organization method of client directory cache in distributed file system | |
| CN109947363A (en) | A kind of data cache method of distributed memory system | |
| CN102541983A (en) | Method for synchronously caching by multiple clients in distributed file system | |
| CN107924291A (en) | Storage system | |
| CN104346373A (en) | Partition log queue synchronization management method and device | |
| CN105740248B (en) | A kind of method of data synchronization, apparatus and system | |
| CN113626431A (en) | LSM tree-based key value separation storage method and system for delaying garbage recovery | |
| CN105653396A (en) | Backup system and backup method thereof | |
| WO2023207492A1 (en) | Data processing method and apparatus, device, and readable storage medium | |
| CN103942301B (en) | Distributed file system oriented to access and application of multiple data types |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220621 Address after: 102600 floor 16, building 5, courtyard 8, KEGU 1st Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing (Yizhuang cluster, high-end industrial area, Beijing Pilot Free Trade Zone) Patentee after: Beijing Aerospace 706 Information Technology Co.,Ltd. Patentee after: Beijing Institute of computer technology and Applications Address before: 100854 No. 51, Yongding Road, Beijing, Haidian District Patentee before: Beijing Institute of Computer Technology and Applications |
|
| TR01 | Transfer of patent right |