KR20170131274A - Method and apparatus for enabling larger memory capacity than physical memory size - Google Patents

Abstract

A method for collecting data stored in a memory related to a de-duplication module is provided. The method comprises the steps of: identifying a logical address of data; identifying a physical line ID (PLID) of the data according to the logical address by searching at least a part of a logical address of a translation table; finding a position of each physical line corresponding to the PLID; collecting the data from each of the physical lines. The collecting step includes a step of copying each hash cylinder to a read cache. Each of the hash cylinders includes: each hash bucket including each of the physical lines; and each reference counter related to each of the physical lines.

Description

[0001] METHOD AND APPARATUS FOR ENABLING LARGER MEMORY [0002] CAPACITY THAN PHYSICAL MEMORY SIZE [

One or more aspects of embodiments of the present invention relate to system memory and storage devices, and more particularly, to memory and storage devices with high capacity low latency.

Typical modern computer applications, such as databases, virtual desktop infrastructure, and data analytics, require large amounts of main memory. As computer systems expand to perform more complex data and storage intensive applications, the demand for larger memory capacity increases proportionally.

A typical random-access memory (RAM) limits the amount of data that can be stored by the physical design of the RAM. For example, an 8 GB DRAM typically has a maximum of 8 GB of data. Future data center applications will also use high capacity low latency memory.

The above-described information disclosed in this Background section is for the purpose of helping to understand the background of the present invention, and may therefore include information that does not constitute prior art.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned technical problems, and it is an object of the present invention to provide a method and apparatus for enabling a memory capacity larger than a physical memory size.

Aspects of embodiments of the present disclosure relate to methods and associated structures that enable memory capacity in RAM that is larger than the physical memory size of the RAM. In accordance with embodiments of the present invention, deduplication algorithms are used to achieve data memory reduction and context addressing. According to embodiments of the present invention, the user data is stored in a hash table indexed by the hash value of the user data.

According to an embodiment of the present invention there is provided a method of retrieving data stored in a memory associated with a dedupe module, the deduplication module including a read cache, wherein the combined data structure comprises a hash table and a reference counter table, wherein each of the hash table and the reference counter table is associated with the combined Wherein the hash table is stored in a plurality of hash cylinders of the data structure, the hash table includes a plurality of hash buckets, each hash bucket includes a plurality of physical lines, each physical line stores data, The reference counter table includes a plurality of reference counter buckets, and each reference counter bucket includes a plurality of reference counters. The method comprising: identifying a logical address of the data; Retrieving at least a portion of the logical address of the translation table to identify a physical line ID (PLID) of the data according to the logical address; Finding a position of each physical line of the plurality of physical lines corresponding to the PLID; And recovering the data from each physical line, wherein the recovering comprises copying each hash cylinder of the plurality of hash cylinders into the read cache, wherein each hash cylinder : Each hash bucket of the plurality of hash buckets, including the respective physical lines; And each reference counter bucket of the plurality of reference counter buckets, each reference counter associated with each physical line.

The method may further include determining, based on the PLID, that the data is stored in the hash table.

The PLID may be generated using a first hash function applied to the data. The PLID may include an address indicating the location of the hash table.

The PLID may include: a first identifier that indicates whether the data is stored in the hash table or in an overflow memory region; A second identifier indicating a row in which the data is stored; And a third identifier indicating a column in which the data is stored.

The combined data structure may further comprise a signature table comprising a plurality of signature buckets, wherein each signature bucket may comprise a plurality of signatures. Each hash cylinder may further comprise a respective signature bucket comprising the plurality of signature buckets and wherein each signature bucket comprises a respective signature associated with each of the physical lines.

The PLID may be generated using a first hash function applied to the data. The PLID may include an address indicating the location of the hash table. The plurality of signatures may be generated using a second hash function that is smaller than the first hash function.

Each reference counter can track the number of duplicates removed for the corresponding data stored in the hash table.

According to an embodiment of the present invention, a method of storing data in a memory associated with a dedupe engine is provided. The method includes: identifying said data to be stored; Determining a first hash value corresponding to a location of the data to be stored in a hash table of the memory using a first hash function; Storing the data at a location of the hash table corresponding to the first hash value; Determining a second hash value corresponding also to a location at which the data is to be stored using a second hash function less than the first hash function; Storing the first hash value in a translation table of the memory; And storing the second hash value in a signature table of the memory.

The method may further comprise incrementing a reference counter of a reference counter table corresponding to the data.

The memory may include: the hash table storing a plurality of data; A conversion table storing a plurality of physical line IDs (PLIDs) generated using the first hash function; The signature table storing a plurality of signatures generated using the second hash function; A reference counter table storing a plurality of reference counters; And an overflow memory region, each reference counter tracks the number of duplicates removed for the corresponding data stored in the hash table.

Each of the plurality of PLIDs may include: a first identifier that indicates whether the data is stored in the hash table or the overflow memory area; A second identifier indicating a row in which the data is stored; And a third identifier indicating a column in which the data is stored.

The hash table, the signature table, and the reference counter table may be combined into a combined data structure. The combined data structure may include a plurality of hash cylinders. Each hash cylinder includes: a hash bucket comprising a plurality of physical lines; A signature bucket comprising respective signatures corresponding to the plurality of physical lines; And a plurality of reference counters corresponding to the plurality of physical lines.

The step of storing the data at a location of the hash table corresponding to the first hash value may include storing the data in the hash bucket corresponding to the first hash value. The step of storing the second hash value in the signature table may include storing the second hash value in the signature bucket corresponding to the hash bucket in which the data is stored.

According to an embodiment of the present invention, a read cache; A dedupe engine for receiving a data retrieval request from the host system; And a memory, said memory comprising: a translation table; And a combined data structure, the combined data structure comprising: a hash table including a plurality of hash buckets; A reference counter table including a plurality of reference counter buckets; And a plurality of hash cylinders, wherein each hash bucket comprises a plurality of physical lines, each physical line storing data, each reference counter bucket comprising a plurality of reference counters, each hash cylinder One of the hash buckets and one of the reference counter buckets, wherein the data retrieval request causes the deduplication engine to: identify the logical address of the data; Searching at least a portion of the logical address of the translation table to identify a physical line ID (PLID) of the data according to the logical address; Find a location of each physical line of the plurality of physical lines, corresponding to the PLID; And recovering the data from each physical line, wherein recovering the data comprises copying each hash cylinder of the plurality of hash cylinders into the read cache, each of the hash cylinders comprising: Each hash bucket of each of the plurality of hash buckets, each containing a respective physical line; A deduplication module is provided that includes a respective reference counter bucket of the plurality of reference counter buckets, each counter reference counter associated with each physical line.

The data retrieval request causes the de-duplication engine to determine, based on the PLID, that the data is stored in the hash table.

The PLID may be generated using a first hash function applied to the data. The PLID may include an address indicating the location of the hash table.

The PLID may include: a first identifier that indicates whether the data is stored in the hash table or in an overflow memory region; A second identifier indicating a row in which the data is stored; And a third identifier indicating a column in which the data is stored.

The combined data structure may further comprise a signature table comprising a plurality of signature buckets, wherein each signature bucket may comprise a plurality of signatures. Each hash cylinder may further include a respective signature bucket comprising the plurality of signature buckets. Each of the signature buckets comprising a respective signature associated with the respective physical line.

The PLID may be generated using a first hash function applied to the data. The PLID may include an address indicating the location of the hash table. The plurality of signatures may be generated using a second hash function that is smaller than the first hash function.

Each reference counter can track the number of duplicates removed for the corresponding data stored in the hash table.

According to an embodiment of the present invention, a host interface; A transmission management unit for receiving data transmission requests from the host system through the host interface; And a plurality of partitions, each partition comprising: a dedupe engine for receiving partition data requests from the transmission manager; A plurality of memory controllers; A memory management unit between the de-duplication engine and the plurality of memory controllers; And a plurality of memory modules, wherein each memory module is connected to one of the plurality of memory controllers.

According to an embodiment of the present invention, there is provided a computer system comprising a read cache and a memory, said memory comprising: a translation table; A hash table including a plurality of hash buckets; A reference counter table including a plurality of reference counter buckets; And a de-duplication engine for identifying V virtual buckets for a first hash bucket of the plurality of hash buckets, wherein each hash bucket comprises a plurality of physical lines, each physical line storing data, Wherein the counter bucket includes a plurality of reference counters, wherein the virtual bucket is other of the plurality of hash buckets adjacent to the first hash bucket, and wherein the virtual buckets are configured such that when the first hash bucket is full, And V is an integer set dynamically based on how full the virtual buckets of the first hash bucket are.

Embodiments of the present invention relate to a plurality of data blocks comprised of the same data to a single stored data block such that duplicate copies of the data blocks can be reduced or eliminated by computer memory, The total amount of unnecessary data copies is reduced. Reducing redundant copies of data can reduce read latency, increase memory bandwidth, and potentially lead to power savings.

These and other features and aspects of the present invention will be appreciated and understood with reference to the specification, the claims, and the accompanying drawings.
1 is a block diagram of a de-duplication module according to an embodiment of the present invention.
2 is a block diagram of a de-duplication module in accordance with another embodiment of the present invention.
3 is a block diagram of a logical view of a deduplication engine in accordance with an embodiment of the present invention.
4 is a block diagram of a logical view of a de-duplication engine including a level-1 conversion table according to an embodiment of the present invention.
5 is a block diagram of a logical view of a de-duplication engine including a level-2 conversion table according to an embodiment of the present invention.
6 is a block diagram of a logical view of a deduplication engine including a dynamic L2 map table, signature and reference counter tables, and a level-2 translation table having an overflow memory area, in accordance with an embodiment of the present invention.
7 is a block diagram of a logical view of a hash cylinder according to an embodiment of the present invention.
8 is a block diagram of a logical view of a combined data structure in accordance with an embodiment of the present invention.
9 is a block diagram of a logical view of a hash bucket associated with virtual buckets and corresponding reference counter buckets in accordance with an embodiment of the present invention.
10 is a flowchart showing a method of recovering data stored in a RAM according to an embodiment of the present invention.
11 is a flowchart showing a method of storing data in a RAM according to an embodiment of the present invention.

Embodiments of the present disclosure are directed to methods and associated structures that enable memory capacity in a memory (e.g., random-access memory (RAM)) that is larger than the physical memory size. In accordance with embodiments of the present invention, deduplication algorithms are used to achieve data memory reduction and context addressing. According to embodiments of the present invention, the user data is stored in a hash table indexed by the hash value of the user data.

While dynamic random access memory (DRAM) technology is aggressively expanding beyond the 20nm process technology to meet this growing demand for memory capacity, techniques such as deduplication may require more than two -3 times or more to increase the virtual memory capacity of the system memory. In addition, embodiments of the present invention may use other types of memory (e.g., flash memory).

Using auxiliary compaction methods, embodiments of the present invention can provide advanced de-duplicated memory and data structures to fully exploit all memory resources to continuously achieve a high deduplication ratio.

Memory with high capacity and low latency is highly required for data center applications. Such memory devices may employ a deduplication scheme as well as a data compression scheme to provide a memory capacity greater than their physical memory size. Deduplicated memory devices can reduce duplicated user data and fully utilize available memory resources to achieve a high deduplication ratio continuously. Also, the deduplication scheme utilized by deduplicated memory devices can achieve effective addressing of de-duplicated data.

Data deduplication (or data duplication elimination) indicates a reduction in redundant data in a memory device, thereby reducing the capacity cost of the memory device. In data de-duplication, a data object / item (e.g., a data file) is divided into one or more data lines / chunks / blocks (lines / chunks / blocks). By associating a plurality of blocks of data comprised of the same data with a block of stored data, redundant copies of the blocks of data can be reduced or eliminated by computer memory, thereby reducing the total number of unnecessary copies of data in the memory device The amount decreases. Reducing redundant copies of data can reduce read latency, increase memory bandwidth, and potentially lead to power savings.

Thus, if duplicate data copies can be reduced to one copy of the data, the total available capacity of the memory device increases while using the same amount of physical resources. The economical use of the resulting memory device can reduce data re-write counts and discard write requests for redundant blocks of data already stored in memory, The life of the memory device can be extended by effectively increasing the write durability.

Related methods of deduplication can use in-memory deduplication techniques, where the deduplication engine is a CPU-centric approach to central processing unit (CPU) ) Or a memory controller (MC). These methods may be used to enable recognition of redundancies of the CPU processor and to enable memory operations (e.g., content lookups, reference count updates, etc.) Typically implement a deduplicated cache (DDC) that works in conjunction with a memory controller to attempt to provide the memory controller. Deduplication methods are also caches for caching translation lines that improve data reads by removing translation fetches from critical paths and are similar to lookaside buffers A direct translation buffer (DTB) can be implemented.

Deduplication is most commonly used for hard drives. However, there is interest in enabling fine grain deduplication in the area of volatile memory such as DRAM.

The detailed description set forth below in connection with the accompanying drawings is intended to enable memory capacity within the RAM (or other memory storage device) to be greater than the physical memory size of the RAM (or other memory storage device) provided in accordance with embodiments of the present invention Are intended as a description of exemplary embodiments of the methods and associated structures for carrying out the invention and are not intended to represent the only forms in which the invention may be constructed or used. The description sets forth the features of the invention in connection with the illustrated embodiments. It is understood, however, that the same or equivalent functions and structures may be achieved by other embodiments, which are intended to be included within the spirit and scope of the present invention. As mentioned elsewhere herein, like reference numerals are intended to depict the same elements or features.

1 is a block diagram of a de-duplication module according to an embodiment of the present invention. 1, a dedupe module 100 according to an embodiment of the present invention includes a bridge 130, a memory controller 140, a host interface (host I / F) 160, a read cache 170, one or more memory modules 180, and a dedupe engine 200.

Bridge 130 provides an interface through which deduplication engine 200 and read cache 170 may communicate with memory controller 140. [ Memory controller 140 may provide an interface to bridge 130 and memory modules 180 for communication. The read cache 170 may be part of the memory modules 180.

In some embodiments, the bridge 180 may not be present. In this case, the memory controller 140 can communicate directly with the deduplication engine 200 and the read cache 170.

Deduplication engine 200 may communicate with the host system via host interface 160 to store data in memory modules 180 or to access data in memory modules 180. Deduplication engine 200 may further communicate with other components of the host system via host interface 160. [

Memory modules 180 may be dual in-line memory module (DIMM) slots or a flash memory for DRAM connections, slots for other types of memory connections, and the like.

2 is a block diagram of a de-duplication module in accordance with another embodiment of the present invention. 2, a dedupe module 150 includes one or more partitions 250, for example, partition 0 (205-0), partition 1 (205-1), etc., a transfer manager 230, and a host interface 162. Each partition 250 includes a de-duplication engine 202, a memory manager 210, one or more memory controllers (e.g., memory controller 0 142, memory controller 1 144, etc.) (E.g., DIMM / Flash 0 182, DIMM / Flash 1 184, etc.).

Each of the deduplication engines 202 may communicate directly with either of the host systems via the transmission management unit 230 or the host interface 162. [ The transmission management unit 230 can communicate with the host system via the host interface 162. [

The transmission management unit 230 may receive data transmission requests from the host system via the host interface 162. [ The transmission management unit 230 may further manage data transmission from one or more partitions 250 of the deduplication module 150 and data transfer from one or more partitions 250 of the deduplication module 150. [ In some embodiments, transmission manager 230 may determine partition 250 to store data to be stored (e.g., stored in RAM). In other embodiments, the transmission manager 230 receives indications from the host system about the partition 250 where the data is to be stored. In some embodiments, the transmission management unit 230 may separate the data received from the host system and send it to two or more partitions.

Deduplication module 150 may communicate with the components of the host system via host interface 162.

Deduplication engine 202 may receive partition data requests from its transmission manager 230 for its respective partition 250. Deduplication engine 202 may further control access and storage of data in memory modules. The memory management unit 210 may determine which one of the one or more memory modules is to be stored or a memory module in which data is to be stored. One or more memory controllers may control the storage or access of data on their respective memory modules.

In some embodiments, the deduplication engine 202 and the memory management section 210 may be implemented as a single memory management section capable of performing the functions of both the memory management section 210 and the deduplication engine 202. [

Each of the one or more memory controllers, memory manager 210, and deduplication engine 202 may be implemented in any suitable hardware (e.g., application-specific integrated circuit (ASIC), firmware FPGA), software, or any suitable combination of software, firmware, and hardware. In addition, de-duplication engine 202 will be described in more detail below.

According to some embodiments, if the memory has a high capacity, the partitions may be used to reduce the translation table size.

3 is a block diagram of a logical view of a deduplication engine in accordance with an embodiment of the present invention. Referring to FIG. 3, the de-duplication engine 200 may include a plurality of tables. The de-duplication engine 200 includes a hash table 220, a translation table 240, signature and reference counter tables 260, and an overflow memory region , 280).

The hash table 220 may comprise a plurality of physical lines (PLs). Each physical line may contain data (e.g., user data). The data in the hash table 220 is deduplicated (i.e., the redundant data is consolidated into one location to reduce space usage of the storage device).

The conversion table 240 includes a plurality of physical line IDs stored therein. Each physical line in the hash table has an associated physical line ID (PLID) stored in the translation table 240. The PLID stored in the conversion table 240 is a logical address to physical address translation. For example, if the deduplication engine 200 needs to find a data location associated with a particular logical address, the deduplication engine 200 can use the conversion table 240 to query the data stored in the logical address And the PLID of the data corresponding to the physical line of the hash table 220 in which the data is stored. The deduplication engine 200 may then access the data stored in the corresponding physical line in the hash table 220.

The PLID may be generated using the first hash function. For example, if data needs to be stored in a hash table, the first hash function is performed on the data to determine a first hash value corresponding to the physical line where the data is to be stored. The first hash value is stored as the PLID of the data.

Each PLID represents the physical location of the targeting data line. The data lines may be in either the hash table 220 or the overflow memory area 280 so that the PLIDs may be locations in the hash table 220 or the overflow memory area 280. [

The hash table 220 may be regarded as a table of row-column structure. In this case, the PLID consists of a region bit, a row bit, and a column bit (see, for example, FIG. 4 and the description thereof). The first hash function may generate a row bit, which is a starting point for looking for a physical line that can be used to store data. Other bits may be determined if an available physical line is found.

If the physical line not found in the hash table 220 is not found in the above step, the data may be written to the overflow memory area 280. In this case, the PLID will be the physical location of the overflow memory area entry.

The second hash value (e.g., signature) of the data calculated using the second hash function is stored in the signature table. The second hash function may be smaller than the first hash function. The first and second hash functions may be any suitable hash function and may be different hash functions.

The signature can be used for a quick comparison between the two data lines. If a new data line is written to the hash table 220, a check may be made to see if the same data line already exists in the hash table. Performing this check can prevent storing the same data multiple times.

If the check is made without using signatures, all data (a whole bucket or an entire virtual bucket) in a particular area of memory is read to detect redundancy. If the checks are made through signatures, then only signatures of the data for a particular area can be read from memory, saving bandwidth.

If there is no matching signature, there is no data line that matches the new data line. Otherwise, if a matching signature is found, the data lines with matching signatures are read from the memory to perform further comparisons, since the signature comparison may be false positive.

Each data line in the hash table has its signature in the signature table and each data line has a corresponding reference counter in the reference counter table.

The reference counter table tracks the number of deduplication counts (e.g., the number of times data has been replicated) for each of the physical lines of the hash table 220. When an instance of deduplicated data is added to a hash table, rather than adding the same new user data as the previously stored user data, the corresponding reference counter of the reference counter table can be incremented, and deduplication from the hash table The corresponding reference counter in the reference counter table may be decremented by one.

In addition, the deduplicated memory (also known as a hash table) consists of physical lines (PLs) which are user data C with a fixed bit width. The length of the default physical line may be 64 bytes, but the present invention is not limited thereto. The PL length may be of different sizes, for example, the PL size may be larger or smaller than 64 bytes. For example, the PL size may be 32 bytes.

A large PL size can reduce the size of the translation table, but can also reduce the amount of redundant data (i.e., the number of deduplications decreases because it needs to match a larger bit pattern). The small PL size can increase the size of the translation table, but can also increase the amount of redundant data (i.e., the number of de-duplications increases).

The translation table stores translations from logical addresses to physical addresses, called physical line IDs (PLIDs). The PLID is generated by a hash function h ₁ (C). Also, for each physical line, there is a signature associated with the physical line stored in the signature table. The signature is a much smaller hash result of the user data, and is generated by the hash function h ₂ (C). The reference counter is also associated with the physical line and stored in the reference counter table. The reference counter counts the number of times the user data (also known as the deduplication rate) matches the PL content.

The hash table, signature table, and reference counter tables all may have the same data structure, but may have different granularity.

Although a plurality of tables are shown as a part of the deduplication module, the present invention is not limited thereto. In accordance with some embodiments of the present invention, a plurality of tables may be stored in a memory (e.g., RAM) in a deduplication module and in accordance with other embodiments, a plurality of tables may be stored in a memory external to the deduplication module , RAM) and controlled by the deduplication module in the manner described herein.

Further descriptions of the above-mentioned features of the invention can be found in U.S. Patent Application (No. 15 / 473,311), the entire contents of which are incorporated herein by reference.

4 is a block diagram of a logical view of a de-duplication engine including a level-1 conversion table according to an embodiment of the present invention. A translation table is a major metadata table that can affect deduplication rate, system capacity, and / or system latency due to its size and the time it takes to use it. 4, logical address 310 may be used by a computer system as the location of data stored in a system memory (e.g., a DRAM).

Logical address 310 may be x bits long, where x is an integer. Logical address 310 may include granularity 314, which is a g bit length, where g is an integer. The granularity 314 may be located in bits from 0 to g-1 of the logical address 310. The logical address 310 may further include a translation table index 312. The translation table index 312 may be xg bits long and may be located in bits from g to x-1 of the logical address 310. [ In some embodiments, if the physical line is 32 bytes long, g is 5 (2 ⁵ = 32) and if the physical line is 64 bytes long, g is 6 (2 ⁶ = 64). In some embodiments, if a virtual capacity of 1 terabyte (terabyte) is supported, x is 40 (2 ⁴⁰ is 1 TB).

The translation table index 312 corresponds to the physical address 320 in the translation table 240. The physical address 320 may include an RGN region bit 322, a row index R_INDX 326, and a column index COL_INDX 328. The region bit (RGN) 322 may be one bit and may indicate whether the data is stored in the hash table 220 or in the overflow memory area 280. The row index R_INDX 326 may be m bits corresponding to M rows (0 to M-1 or 0 to 2 ^m -1) in the hash table 220. The column index (COL_INDX, 328) may be n bits corresponding to N columns (0 to N-1 or 0 to 2 ⁿ -1) in the hash table 220. M, N, m, and n are integers. According to certain embodiments, the hash table is 128GB (2 ^37), is g = 6, m = 26, n = 5, M = 2 26, and N = 2 ^5.

The overflow memory area 280 also stores data that is not placed in the hash table.

5 is a block diagram of a logical view of a de-duplication engine including a level-2 conversion table according to an embodiment of the present invention. Conversion tables are key metadata tables that can affect deduplication rates, system capacity, and system latency. 5, the translation table includes level-2, page index table 242, and level 2 (L2) map table 244.

The logical address 310 'may be used by a computer system as the location of data stored in a memory (e.g., RAM). The length of the logical address 310 'may be x bits long, where x is an integer. The logical address 310 'may include a granularity 314' that is a g bit length, where g is an integer. The granularity 314 'may be located in bits from 0 to g-1 of the logical address 310'. The logical address 310 'may further include a page entry 318 and a page index 316. The page entry 318 may be 12-g bits long and may be located in bits g through 11 of the logical address 310 '. The page index 316 may be x-12 bits long and may be located in bits 12 to x-1 of the logical address 310 '. In some embodiments, if the physical line is 32 bytes long, g is 5 (2 ⁵ = 32) and if the physical line is 64 bytes long, g is 6 (2 ⁶ = 64). In some embodiments, if a virtual capacity of 1 TB is supported, x is 40 (2 ⁴⁰ is 1 TB).

The page index 316 corresponds to the page in the page index table 242. The page in the page index table 242 corresponds to the entry 0 position in the L2 map table 244. [ The page entry 318 indicates which entry after entry 0 stores the physical address 320 'of the stored data corresponding to the logical address 310'.

In other words, the page index 316 is associated with a set of L2 map entries and a page entry 318 designated with an entry of that set. The page index 316 leads to the first entry in the set, and the page entry 318 shows which particular entry of that set of entries contains the physical address 320 '. Each page in the page index table 242 may include a region bit RGN. The area bits RGN 322 'may be one bit and may indicate whether the data is stored in the hash table 220' or the overflow memory area 280 '.

The physical address 320 'may include a row index R_INDX 326' and a column index COL_INDX 328 '. The row index R_INDX 326 'may be m bits corresponding to M rows (0 to M-1 or 0 to 2 ^m -1) in the hash table 220'. The column index (COL_INDX, 328 ') may be n bits corresponding to N columns (0 to N-1 or 0 to 2 ⁿ -1) in the hash table 220'. M, N, m, and n are integers. According to certain embodiments, the hash table is 128GB (2 ^37), is g = 6, m = 26, n = 5, M = 2 26, and N = 2 ^5.

The overflow memory area 280 also stores data that is not placed in the hash table.

6 is a block diagram of a logical view of a de-duplication engine including a dynamic L2 map table and a level-2 translation table having an overflow memory area, in accordance with an embodiment of the present invention. Referring to FIG. 6, the level-2 conversion table may create additional space for the overflow memory area.

In some embodiments, the sizes of the signature and reference counter tables 260 'and the page index table 242' are fixed, but the sizes of the L2 map table 244 'and the overflow memory area 280' to be.

As the sizes of L2 map table 244 'and overflow memory area 280 " increase, they increase towards each other. In this way, the storage space can be efficiently used by allowing either the L2 map table 244 'or the overflow memory area 280 " to grow toward an unused space.

7 is a block diagram of a logical view of a hash cylinder according to an embodiment of the present invention. 8 is a block diagram of a logical view of a combined data structure in accordance with an embodiment of the present invention. 7 and 8, the signature table, the reference counter table, and the hash table are stored in the hash cylinders (e.g., the combined structure 600 (e.g., the combined structure 600 or the combined table 600) (E.g., a hash bucket (i)) in the first, second, third, fourth, fifth, Each hash cylinder 500 includes a hash table hash bucket 560 (e.g., a hash bucket 560-i), a signature table signature bucket 520 (e.g., signature bucket 520-i) And a reference counter bucket 540 (e.g., reference counter bucket (i)) of the reference counter table.

The hash bucket 560 includes a plurality of entries (e.g., entries (0) through (N-1)) of physical lines.

The signature bucket 520 includes a plurality of signatures corresponding to data stored in physical lines in the hash bucket 560 of the same hash cylinder 500.

The reference counter bucket 540 includes a plurality of reference counters corresponding to the number of times data stored in the physical lines in the hash bucket 560 of the same hash cylinder 500 are deduplicated.

In other words, the hash table is divided into a plurality of hash buckets 560, and each hash bucket 560 includes a plurality of entries. The signature table is divided into a plurality of signature buckets 520, and each signature bucket 520 includes a plurality of signatures. The reference counter table is divided into a plurality of reference counter buckets 540, and each reference counter bucket 540 includes a plurality of reference counters.

The combined data structure 600 is configured to be placed in the hash cylinder 500 together with one hash bucket 560, one signature bucket 520, and one reference counter bucket 540. In accordance with some embodiments of the present invention, the buckets are arranged in the following order: a first signature bucket 520-0, a first reference counter bucket 540-0, a first hash bucket 560-0, 2 signature bucket 520-1, a second reference counter bucket 540-1, a second hash bucket 560-1, and so on.

In this arrangement, the first signature bucket 520-0 includes signatures associated with the data stored in the first hash bucket 560-0 and the first reference counter bucket 540-0 includes the first hash bucket 560- 0 < / RTI > In addition, the second signature bucket 520-1 includes signatures associated with the data stored in the second hash bucket 560-1 and the second reference counter bucket 540-1 includes the second hash bucket 560-1, Lt; RTI ID = 0.0 > and / or < / RTI > In addition, the first cylinder 500-0 includes a first signature bucket 520-0, a first reference counter bucket 540-0, and a first hash bucket 560-0, -1 includes a second signature bucket 520-1, a second reference counter bucket 540-1, and a second hash bucket 560-1.

In this manner, each hash cylinder 500 includes data and signatures associated with data stored in the same hash bucket 500 and reference counters.

When a request is made for data stored in the hash cylinder 500-i of the combined data structure 600, the entire hash cylinder 500-i is copied to the read cache 170 '. Since the entire hash cylinder 500-i is copied into the read cache 170 ', the time required to retrieve all of the requested data, the signature (or each signature), and the corresponding reference counter (or each reference counter) .

According to some embodiments, the read data cache may be the same size as the hash cylinder.

Also, if the deduplication engine determines that data is already present in the hash table (to avoid duplication), the entire hash cylinder 500 may be copied to the read cache 170 '. Because the deduplication engine has access to signatures, reference counters, and data when deciding whether deduplication is possible and storing the data, having the read cache copy the entire hash cylinder can reduce the access time, .

In other words, a hash cylinder 500, which is an integral unit of hash entries, signatures, and reference counter entries, can be created to improve latency and performance. The integrated hash cylinder 500 can reduce system memory access cycles and improve system latency. A compacted data structure can reduce the number of memory accesses. Each hash cylinder 500 contains all the information necessary for the deduplication engine to perform the computation. The combined data structure 600 may also facilitate caching.

9 is a block diagram of a logical view of a hash bucket associated with virtual buckets and corresponding reference counter buckets in accordance with an embodiment of the present invention. Referring to FIG. 9, each hash bucket 560 'may be associated with one or more virtual buckets (VBs, e.g., VB (0) to VB (V-1)). Each hash bucket 560 'may include N ways (e.g., WAY (0) to WAY (N-1)).

Unlike the hash tables of related fields, the hash tables of the present embodiment each include a plurality of virtual hash buckets or virtual buckets, and the virtual buckets comprise a plurality of physical hash buckets or physical buckets. Hereinafter, the term " physical bucket " will refer to the previously described hash buckets and will be used to distinguish the virtual buckets from the previously described hash buckets.

Each virtual bucket may include some of the physical buckets of the hash table. However, it should be noted that other of the virtual buckets may share one or more physical buckets. As will be described below, additional dimensions are added to the hash table, using virtual buckets in accordance with embodiments of the present invention. Thus, greater flexibility in arranging and arranging data can be provided, thereby increasing the efficiency of the deduplication DRAM system and increasing the compression ratio.

This embodiment allows a block of data stored in one of the hash buckets to be moved into the corresponding virtual bucket or to another physical bucket in order to secure other physical buckets shared by other virtual buckets, Lt; / RTI > to increase the flexibility of the data placement of the virtual buckets. By ensuring space within the hash table, deduplication can be achieved by eliminating useless / duplicated data. That is, by using the virtual buckets according to the embodiments of the present invention, there is no strict restriction due to hashing of the lines of data to a limited corresponding location using a hash function, Quot; physical bucket, " which represents a physical bucket in the same virtual bucket that initially contains the intended (but occupied) physical hash bucket.

By way of example, content (e.g., a data line) is disposed in one of the physical buckets. Instead of requiring the data line to be placed in a physical bucket if the data line is placed in the first physical bucket, this embodiment is also a virtual bucket that is larger than a single physical bucket and includes physical buckets as well as other physical buckets It is possible. That is, the virtual bucket includes the sum of adjacent or contiguous physical buckets aligned in the hash table.

Thus, the virtual buckets may cause the data blocks to move within the hash table to free up space for future write operations.

For further discussion of virtual buckets, see U.S. Patent Application No. 15 / 162,512, filed May 23, 2016, and U.S. Patent Application No. 15 / 162,517, filed May 23, 2016, , The entire contents of which are incorporated herein by reference.

Also, virtual buckets may have dynamic heights or sizes. Having a dynamic virtual bucket height (VBH) can improve memory usage in a limited latency impact.

The number of virtual buckets associated with the physical bucket is indicated by the height index of the virtual bucket (VB). The virtual bucket height information is stored in the last reference counter of the reference counter bucket 540 'associated with the hash bucket 560'. A portion of the bits of the reference counter is used as the VB height index (e.g., VBH [1: 0]).

Using the hash bucket (i) as an example, if the VB height is V, the virtual buckets of the hash bucket (i) may represent a hash bucket (i + V) from the hash bucket (i + 1). When the hash bucket (i) is full, the deduplication engine will put user data in the virtual buckets.

A flag (flag, a portion of one RC bit, e.g., the last RC counter of the hash bucket M) indicates how many virtual buckets are currently being used by the hash bucket (i). In this way, the latency can be reduced because there is no need to retrieve more virtual buckets than are needed. Virtual buckets in related fields use a fixed VB height. Using a fixed virtual bucket height, the search logic retrieves all virtual buckets, regardless of the number of virtual buckets actually used by the hash bucket i, which can result in increased wait times.

Virtual buckets do not require additional memory space. They use unused entries near the hash buckets. For example, for a hash bucket (i + 1), its virtual buckets represent a hash bucket (i + V '+ 1) from a hash bucket (i + 2).

In addition, if the virtual buckets (e.g., hash bucket (i + 1) to hash bucket (i + V)) of the hash bucket i are full, then the deduplication engine according to the embodiment of the present invention, Increase the height (V) of the virtual bucket to use the space available nearby. The heights of the virtual buckets in the relevant field can not be increased because they have been predetermined (rather than dynamic). Thus, if the virtual buckets (e.g., hash buckets (i + 1) to hash buckets (i + V) of the hash bucket i) are full, Can not increase the height (V).

In addition, by dynamically adjusting the height of the virtual buckets, if the deduplication engine determines that the data is already in the hash table (to avoid duplication), the deduplication engine may replace the virtual buckets with a predetermined number of virtual buckets You only need to check the virtual buckets you are using. This can reduce the access time and increase the overall operation speed.

10 is a flowchart showing a method of recovering data stored in a RAM according to an embodiment of the present invention. Although FIG. 10 is illustrated using RAM, the present invention is not so limited and any other suitable memory type may be used with these methods.

Referring to FIG. 10, the CPU of the computer system may request data stored in the RAM. The CPU can provide an address for the location of data in RAM. The present invention is not limited thereto, for example, other components may request data from the RAM and provide a logical address.

A method for retrieving data stored in a RAM according to an embodiment of the present invention includes identifying a logical address of data stored in the RAM (operation 1000). The logical address may correspond to the location of the translation table.

The method further includes identifying the PLID (physical line ID) of the data according to the logical address by searching the logical address of the conversion table (step 1010).

The method further includes determining, based on the PLID, whether the data is stored in a hash table of RAM or in an overflow memory area of RAM (step 1020).

If the data is stored in the hash table, the method further includes the step of finding the position of the physical line of the hash table corresponding to the PLID (step 1030) and the step of recovering data from the physical line of the hash table (step 1040). The step of recovering the data includes recovering the data from the signature table and the reference counter table.

If the data is stored in the overflow memory, the method includes locating the physical line of the overflow memory area corresponding to the PLID (step 1050) and recovering the data from the physical line of the overflow memory area (step 1060) .

The PLID may be generated using the first hash function applied to the data. The PLID may include an address that indicates the location of the hash table of the RAM or the overflow memory area of the RAM.

A PLID may include: a first identifier (e.g., an RGN reference in FIG. 4) that indicates whether data is stored in the hash table or overflow memory area; A second identifier (e.g., see R_INDX in FIG. 4) that represents the row in which the data is stored; And a third identifier (e.g., COL_INDX in FIG. 4) indicating the column in which the data is stored.

The method may further include recovering a signature associated with the data from the signature table.

The RAM includes a hash table storing a plurality of data; A conversion table storing a plurality of PLIDs generated using the first hash function; A signature table storing a plurality of signatures generated using a second hash function smaller than the first hash function; A reference counter table including a plurality of reference counters, each reference counter tracks the number of deduplication counts for the corresponding data stored in the hash table; And an overflow memory area.

The hash table, signature table, and reference counter table may be combined into a combined data structure. The combined data structure may comprise a plurality of hash cylinders and each hash cylinder may comprise: a hash bucket comprising a plurality of physical lines; A signature bucket comprising respective signatures corresponding to a plurality of physical lines; And a reference counter bucket comprising respective reference counters corresponding to the plurality of physical lines.

Recalling data from a physical line or overflow memory area may include copying the entire hash cylinder containing the physical line, the signature, and the corresponding reference counter into the read cache.

11 is a flowchart showing a method of storing data in a RAM according to an embodiment of the present invention. Although FIG. 11 is shown using RAM, the present invention is not so limited and any other suitable memory type may be used with these methods.

Referring to FIG. 11, the CPU of the computer system may request that data be stored in the RAM. The CPU can provide data to be stored in the RAM. The present invention is not limited thereto, and for example, other components can request and store data in the RAM.

A method of storing data in a RAM according to embodiments of the present invention includes identifying data to be stored in RAM (step 1100).

The method further includes determining a first hash value corresponding to a location where the data should be stored in the hash table of the RAM using the first hash function (step 1110).

The method further includes storing data at a location of the hash table corresponding to the first hash value (step 1120).

The method further includes determining a second hash value corresponding to the location where the data should be stored using the second hash function (step 1130). The second hash function may be smaller than the first hash function.

The method further includes storing the first hash value in a translation table (step 1140).

The method further includes storing the second hash value in a signature table (step 1150).

The method may further include, within the reference counter table, incrementing a reference counter corresponding to the data.

RAM may include: a hash table storing a plurality of data; A conversion table storing a plurality of PLIDs generated using the first hash function; A signature table storing a plurality of signatures generated using the second hash function; A reference counter table storing a plurality of reference counters, each reference counter tracks the number of times of deduplication for the corresponding data stored in the hash table; And overflow memory areas.

Each of the PLIDs may include: a first identifier (e.g., see RGN in FIG. 4) that indicates whether data is stored in a hash table or an overflow memory area; A second identifier (e.g., see R_INDX in FIG. 4) that represents the row in which the data is stored; And a third identifier (e.g., COL_INDX in FIG. 4) indicating the column in which the data is stored.

The hash table, signature table, and reference counter table may be combined into a combined data structure. The combined data structure may include a plurality of hash cylinders. Each hash cylinder may include: a hash bucket comprising a plurality of physical lines; A signature bucket comprising respective signatures corresponding to a plurality of physical lines; And a reference counter bucket comprising respective reference counters corresponding to the plurality of physical lines.

Storing data at a location of the hash table corresponding to the first hash value may include storing the data in a hash bucket corresponding to the first hash value. Storing the second hash value in the signature table may include storing the second hash value in the signature bucket corresponding to the hash bucket where the data is stored.

Thus, the embodiments herein relate to methods and association structures that enable memory capacity in a memory (e.g., random-access memory (RAM)) that is larger than the physical memory size. In accordance with embodiments of the present invention, de-duplication is used to achieve data memory reduction and context addressing. According to embodiments of the present invention, the user data is stored in a hash table indexed by the hash value of the user data.

Although the terms first, second, third, etc. are used herein to describe various elements, components, regions, layers and / or sections, these terms, components, Or sections are not intended to be limited by these terms. These terms are used to distinguish one element, component, region, layer or section from another element, element, region, layer, or section. Thus, a first component, component, region, layer, or section described below may refer to a second component, component, region, layer, or section without departing from the spirit and scope of the present invention.

 In accordance with embodiments of the invention described herein, an associated device or component (or related devices or components, e.g., a deduplication engine) may be any suitable hardware (e.g., ASIC) (E. G., A DSP or FPGA), software, or any suitable combination of software, firmware, and hardware. For example, the various components of the associated device (s) may be formed on an integrated circuit (IC) chip or separate IC chips. The various components of the associated device (s) may also be implemented on a flexible printed circuit film (FPCF), a tape carrier package (TCP), a printed circuit board (PCB), or a combination of one or more circuits and / Can be formed on the same substrate. In addition, the various components of the associated device (s) may be implemented on one or more processors and may be, in one or more computing devices, configured to execute computer program instructions and interact with other system components to perform the various functions described herein. Lt; / RTI > may be a working process or thread. Computer program instructions are stored in a memory that can be implemented in a computing device using standard memory devices, such as, for example, RAM. The computer program instructions may also be stored on other non-volatile computer readable media, such as, for example, CD-ROMs, flash drives, and the like. Those skilled in the art will also appreciate that the functionality of the various computing devices may be combined or integrated into a single computing device or the functionality of a particular computing device may be implemented within one or more other computing devices without departing from the spirit and scope of the exemplary embodiments of the present invention ≪ / RTI >

Also, when referring to an element, element, region, layer, and / or section being "between" two elements, elements, regions, layers, and / Element, element, element, region, layer, and / or section between elements, elements, regions, layers, and / or sections of one or more intermediate elements, elements, , ≪ / RTI > and / or sections.

The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting of the invention. The singular forms as used herein are intended to include the plural forms, unless the context clearly indicates otherwise. The terms "comprises" and "comprising", when used in this specification, specify specified features, integers, steps, operations, elements, and / or components but may include one or more other features, Steps, operations, elements, and / or components described in connection with the embodiments disclosed herein.

As used herein, the term "and / or" includes any and all combinations of one or more of the listed listed items. Expressions such as "at least one," "one, " and" select from "modify the entire list of elements and do not modify individual elements of the list, Further, the use of "can" when describing embodiments of the present invention represents "one or more embodiments of the present invention ". Also, the "exemplary" is intended to be illustrative or illustrative.

As used herein, "use", "use", and "used" may be considered synonymous with the terms "use", "use", and "used", respectively.

The features described in connection with one or more embodiments of the invention may be used with the features of other embodiments of the invention. For example, the features described in the first embodiment may be combined with the features described in the second embodiment to form the third embodiment, although the third embodiment is not specifically described herein.

Those skilled in the art will also appreciate that the process may be executed through hardware, firmware (e.g., via an ASIC), or any combination of software, firmware, and / or hardware. In addition, the order of the steps of the process is not fixed, but may be changed in any desired order recognized by those skilled in the art. The altered order may include all steps or portions of steps.

Although the present invention has been described in connection with specific embodiments thereof, it will be apparent to those skilled in the art that modifications will be apparent to those skilled in the art, without departing from the scope and spirit of the invention. In addition, those skilled in the art will appreciate that the invention itself as set forth herein will suggest solutions to other problems and adaptations to other applications. It is the intent of the applicants to include all such uses of the invention and variations and modifications thereof which may achieve embodiments of the invention selected for the purpose of disclosure without departing from the spirit and scope of the invention. Accordingly, the embodiments of the present invention are to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims and equivalents thereof.

100: Deduplication module
130: Bridge
140: Memory controller
160: Host interface
170: Read Cache
180: DIMM / Flash
200: Deduplication engine

Claims (22)

A method for recovering data stored in a memory associated with a dedupe module, the deduplication module comprising a read cache, the memory including a translation table and a combined data structure Wherein the combined data structure comprises a hash table and a reference counter table, wherein each of the hash table and the reference counter table comprises a plurality of hash cylinders of the combined data structure, cylinder, the hash table comprising a plurality of hash buckets, each hash bucket comprising a plurality of physical lines, each physical line storing data, the reference counter table comprising a plurality of reference counter buckets, Each reference counter bucket comprising a plurality of reference counters, the method comprising:
Identifying a logical address of the data;
Retrieving at least a portion of the logical address of the translation table to identify a physical line ID (PLID) of the data according to the logical address;
Finding a position of each physical line of the plurality of physical lines corresponding to the PLID; And
And recovering the data from each physical line,
Wherein the recovering step includes copying each hash cylinder of the plurality of hash cylinders into the read cache,
Each of the hash cylinders comprising:
A respective hash bucket of each of the plurality of hash buckets, the respective hash buckets comprising the respective physical lines; And
Each reference counter bucket of the plurality of reference counter buckets comprising a respective reference counter associated with each physical line. The method according to claim 1,
Further comprising determining based on the PLID that the data is stored in the hash table. The method according to claim 1,
The PLID is generated using a first hash function applied to the data, and
And the PLID includes an address indicating the location of the hash table. The method of claim 3,
The PLID is:
A first identifier indicating whether the data is stored in the hash table or in an overflow memory region;
A second identifier indicating a row in which the data is stored; And
And a third identifier indicating a column in which the data is stored. The method according to claim 1,
Wherein the combined data structure further comprises a signature table comprising a plurality of signature buckets, each signature bucket comprising a plurality of signatures, and
Wherein each hash cylinder further comprises a respective signature bucket comprising the plurality of signature buckets, wherein each signature bucket comprises a respective signature associated with the respective physical line. 6. The method of claim 5,
The PLID is generated using a first hash function applied to the data, and
Wherein the PLID includes an address indicating a location of the hash table,
Wherein the plurality of signatures are generated using a second hash function less than the first hash function. The method according to claim 1,
And each reference counter tracks the number of duplicates removed for the corresponding data stored in the hash table. A method of storing data in a memory associated with a dedupe engine,
Identifying said data to be stored;
Determining a first hash value corresponding to a location of the data to be stored in a hash table of the memory using a first hash function;
Storing the data at a location of the hash table corresponding to the first hash value;
Determining a second hash value corresponding also to a location at which the data is to be stored using a second hash function less than the first hash function;
Storing the first hash value in a translation table of the memory; And
Storing the second hash value in a signature table of the memory. 9. The method of claim 8,
Further comprising increasing a reference counter of a reference counter table corresponding to the data. 9. The method of claim 8,
The memory comprising:
The hash table storing a plurality of data;
A conversion table storing a plurality of physical line IDs (PLIDs) generated using the first hash function;
The signature table storing a plurality of signatures generated using the second hash function;
A reference counter table storing a plurality of reference counters; And
An overflow memory region,
And each reference counter tracks the number of duplicates removed for the corresponding data stored in the hash table. 11. The method of claim 10,
Wherein each of the plurality of PLIDs comprises:
A first identifier indicating whether the data is stored in the hash table or in the overflow memory area;
A second identifier indicating a row in which the data is stored; And
And a third identifier indicating a column in which the data is stored. 11. The method of claim 10,
The hash table, the signature table, and the reference counter table are merged into a combined data structure, and
The combined data structure comprising a plurality of hash cylinders, each hash cylinder comprising:
A hash bucket comprising a plurality of physical lines;
A signature bucket comprising respective signatures corresponding to the plurality of physical lines; And
And a reference counter bucket comprising respective reference counters corresponding to the plurality of physical lines. 13. The method of claim 12,
Wherein storing the data at a location of the hash table corresponding to the first hash value comprises storing the data in the hash bucket corresponding to the first hash value,
Wherein storing the second hash value in a signature table of the memory comprises storing the second hash value in the signature bucket corresponding to the hash bucket where the data is stored. Read cache;
A dedupe engine for receiving a data retrieval request from the host system; And
Memory,
The memory comprising:
A translation table; And
Include a combined data structure,
The combined data structure comprises:
A hash table containing a plurality of hash buckets;
A reference counter table including a plurality of reference counter buckets; And
A plurality of hash cylinders,
Wherein each hash bucket comprises a plurality of physical lines, each physical line storing data, each reference counter bucket comprising a plurality of reference counters, each hash cylinder having one of the hash buckets and the reference counter buckets ≪ / RTI >
Wherein the data retrieval request causes the deduplication engine to:
Identify a logical address of the data;
Searching at least a portion of the logical address of the translation table to identify a physical line ID (PLID) of the data according to the logical address;
Find a location of each physical line of the plurality of physical lines, corresponding to the PLID; And
And causing the data to be recovered from the respective physical lines, wherein recovering the data comprises copying each hash cylinder of the plurality of hash cylinders into the read cache,
Each of the hash cylinders comprising:
A respective hash bucket of each of the plurality of hash buckets, the respective hash buckets comprising the respective physical lines; And
Each reference counter bucket of each of the plurality of reference counter buckets, each reference counter associated with each physical line. 15. The method of claim 14,
Wherein the data retrieval request further causes the deduplication engine to determine, based on the PLID, that the data is stored in the hash table. 15. The method of claim 14,
The PLID is generated using a first hash function applied to the data, and
And the PLID includes an address indicating the location of the hash table. 17. The method of claim 16,
The PLID is:
A first identifier indicating whether the data is stored in the hash table or in an overflow memory region;
A second identifier indicating a row in which the data is stored; And
And a third identifier indicating a column in which the data is stored. 15. The method of claim 14,
Wherein the combined data structure further comprises a signature table comprising a plurality of signature buckets, each signature bucket comprising a plurality of signatures, and
Wherein each hash cylinder further comprises a respective signature bucket comprising the plurality of signature buckets, wherein each signature bucket comprises a respective signature associated with each of the physical lines. 19. The method of claim 18,
The PLID is generated using a first hash function applied to the data, and
Wherein the PLID includes an address indicating a location of the hash table,
Wherein the plurality of signatures are generated using a second hash function smaller than the first hash function. 15. The method of claim 14,
And each reference counter tracks the number of duplicates removed for the corresponding data stored in the hash table. Host interface;
A transmission management unit for receiving data transmission requests from the host system through the host interface; And
A plurality of partitions,
Each partition is:
A dedupe engine receiving the partition data requests from the transmission management unit;
A plurality of memory controllers;
A memory management unit between the de-duplication engine and the plurality of memory controllers; And
A plurality of memory modules,
Each memory module being coupled to one of the plurality of memory controllers. A read cache and memory,
The memory comprising:
A translation table; And
A hash table including a plurality of hash buckets;
A reference counter table including a plurality of reference counter buckets; And
A de-duplication engine for identifying V virtual buckets for a first hash bucket of the plurality of hash buckets,
Wherein each hash bucket comprises a plurality of physical lines, each physical line storing data, each reference counter bucket comprising a plurality of reference counters, said virtual bucket comprising a plurality of hash buckets adjacent to said first hash bucket, And the virtual buckets store a portion of the data of the first hash bucket when the first hash bucket is full and V is set dynamically based on how full the virtual buckets of the first hash bucket are A de-duplication module that is an integer that is an integer.

Images