JP7058132B2 - Systems and methods for maximized deduplication memory - Google Patents

Description

The present invention relates generally to memory, and more particularly to systems and methods for maximizing deduplication memory.

Deduplicated (or deduplicated) memory provides a more efficient mechanism for storing data. In traditional memory solutions, each data object is written to a unique location in memory. The same data object is stored in any number of locations in memory, each being a separate copy of that data object. The memory system has no way of identifying or preventing such repeated storage of data. Such repetitive storage of data is useless for large data objects. Deduplication memory, which stores only one copy of any data object, attempts to solve this problem.

Some deduplication memory uses a hash table to store data objects. However, the hash table is only incremented by a mechanism that doubles its size. Such a large granularity often leaves a large portion of the memory that cannot be used as the deduplication memory and is simply treated as an overflow area. Since the memory in the overflow area is not deduplicated, the overall deduplication ratio will be reduced if a large portion of the memory cannot be deduplicated.

Therefore, there is still a need for a method to increase the proportion of memory targeted for deduplication.

U.S. Pat. No. 9639274 U.S. Patent Application Publication No. 2015/0370835 U.S. Patent Application Publication No. 2017/0161329

The present invention has been made in view of the above-mentioned conventional problems, and an object of the present invention is to provide a system and a method for increasing a portion of memory to be deduplicated.

A memory system according to an aspect of the present invention made to achieve the above object includes a memory for storing data, a predetermined number of buckets and a first number of ways stored in the memory, and a first of two. A large hash table containing the first part of the memory containing the first part of the memory, which is a power of, and a second number of 2 buckets stored in the memory containing a predetermined number of buckets and a second number of ways . A small hash table containing a second part of the memory containing a second number of bytes that is a power , an overflow area stored in the memory containing the third part of the memory, and a logical address as an area identifier and a physical address. It is characterized by comprising a conversion table for mapping to a PLID (Physical Line Idea) including the above.

A method of a memory system according to an aspect of the present invention made to achieve the above object uses a step of receiving a logical address from a processor and a conversion table to convert the logical address to a PLL including an area identifier and a physical address. The stage of mapping to (Physical Line Idea), the stage of determining whether or not the physical address exists in the large hash table, the small hash table, or the overflow area in the memory by using the area identifier, and the physical. It is characterized by having a stage of accessing the data in the memory by using an address.

A computer-readable recording medium according to an aspect of the present invention made to achieve the above object records a program for causing a computer to execute the following method, and the method is a step of receiving a logical address from a processor. , A step of mapping the logical address to a PLID (Physical Line Identity) including an area identifier and a physical address using a conversion table, and a large hash table and a small hash table in memory using the area identifier. Alternatively, it is characterized by having a step of determining whether or not the physical address exists in the overflow area, and a step of accessing the data in the memory by using the physical address.

According to the present invention, deduplicated (or deduplicated) memory provides a more effective mechanism for storing data. It also improves memory usage and reduces the deduplication ratio required for effective deduplication.

It is a figure which shows the machine which operates for using the deduplication memory by one Embodiment of this invention. It is a figure which shows the further detail of the machine shown in FIG. It is a figure which shows the use of the conventional hash table for the deduplication memory in the machine of FIG. It is a figure which shows the use of the expandable hash table by one Embodiment of this invention. It is a figure which shows the use of the expandable hash table by one Embodiment of this invention. It is a figure which shows the use of the conversion table of FIG. 4 to map a logical address to various memory destinations. It is a flowchart which shows an example of the procedure which uses the expandable hash table of FIG. 4 for the deduplication memory by one Embodiment of this invention. It is a flowchart which shows an example of the procedure which uses the expandable hash table of FIG. 4 for the deduplication memory by one Embodiment of this invention. It is a flowchart which shows an example of the procedure for discriminating the PLID (Physical Line Identifier) with respect to the logical address in the read request of the memory by one Embodiment of this invention. It is a flowchart which shows an example of the procedure for discriminating the PLID with respect to the logical address in the write request of the memory by one Embodiment of this invention. It is a flowchart which shows an example of the procedure for discriminating the PLID with respect to the logical address in the write request of the memory by one Embodiment of this invention. It is a flowchart which shows an example of the procedure for discriminating the PLID with respect to the logical address in the write request of the memory by one Embodiment of this invention. It is a flowchart which shows an example of the procedure for determining whether or not to increase the size of a small hash table by one Embodiment of this invention.

Hereinafter, specific examples of embodiments for carrying out the present invention will be described in detail with reference to the drawings. In the detailed description described below, a number of specific examples are presented so that the concept of the present invention can be fully understood. However, an ordinary engineer in the relevant technical field can carry out the present invention without such a specific case. Widely known methods, devices, components, circuits, and networks are not described in detail in order not to unnecessarily obscure aspects of the invention.

The first, second, and the like are used herein to describe the various components, but these components are not limited by such terms. For example, to the extent that it does not deviate from the technical scope of the present invention, the first module is referred to as a second module, and similarly, the second module is referred to as a first module.

The terms used in the description of the present invention are used only for the purpose of describing a particular embodiment and do not limit the technical idea of the present invention. Singular expressions include multiple expressions unless explicitly stated in the context. The term "and / or" includes any and possible combinations of one or more associated entries. The terms "contains" and / or "contains" specify the presence of the described properties, integers, steps, actions, elements, and / or components, one or more other properties, integers, Does not rule out the presence or addition of stages, actions, elements, components, and / or groups thereof. The components and properties shown in the drawings are not necessarily proportional to the actual ratio.

In a conventional hash table, the size of the hash table is represented by m × n, where m is a hash bucket (or low) number and n is a way (or column) number. For example, the hash table has 2 ^m = 2 ²⁶ hash buckets and 2 ⁿ = ²⁵ ways.

When the traditional hash table grows, the increase in hash table size is twice the current size (since N increases the exponent, the dimension of the hash table is doubled). The number of hash buckets does not change, only the number of ways. Therefore, it may not be possible to double the size of the hash table, depending on the amount of memory available and the size of the hash table. This fact leaves a large portion of memory that is not deduplicated, and this portion is used as an overflow area.

As used herein, a scalable hash table includes a large hash table, which is a conventional hash table, and a small hash table that includes the same number of hash buckets but has a smaller number of ways. (Little Hash Table) is included. So, for example, if 2 ^m = 2 ²⁶ and 2 ⁿ = ²⁵ (and instead m = 26 and n = 5) for a large hash table, then the size of the small hash table is 2 ^m = 2 ²⁶ and 2 ^n' , where n'is any number between 1 and n-1. The extensible hash table adjusts the hash table size with finer granularity to maximize the deduplication memory size, providing flexibility to achieve high deduplication ratios. .. The names "large hash table" and "small hash table" are merely to clarify which hash table is referenced. Hash tables are simply referred to simply as "first hash tables" and "second hash tables" without losing any functionality.

The mapping of logical addresses to physical memory (also known as physical line ID, or PLID (Physical Line ID)) is managed by a translation table. User data (also known as a physical line or PL (Physical Line)) is stored in one of a large hash table, a small hash table, and an overflow area.

The translation table entry includes an area indicating whether the user data is present in one of the large hash tables or the small hash table or overflow area. So, for example, if this area stores a value of 0, the user data will be found in a large hash table. Otherwise, user data will be found in a small hash table or overflow area.

If the user data is not stored in a large hash table, the translation table entry also contains a subregion that indicates which of the smaller hash table and the overflow area stores the user data. So, for example, if the subregion stores the value 0, the user data is found in a small hash table. Otherwise, user data will be found in the overflow area.

FIG. 1 shows a machine operating to use the deduplication memory according to an embodiment of the invention. FIG. 1 shows the machine 105. The machine 105 can be a desktop computer, a laptop computer, a server (one of a stand-alone server or a rack server), or any other embodiment of the invention that can benefit from it. Includes equipment. However, it is not limited to these. Machine 105 includes specialized portable computing devices, tablet computers, smartphones, and other computing devices. The machine 105 runs any desired application program. A database application is a good example, but embodiments of the invention extend to any desired application.

The machine 105 includes a processor 110, a memory 115, and a storage device 120, regardless of the specific form. The processor 110 is any variety of processors. For example, Intel® Xeon®, Celeron®, Itanium®, or Atom® processor, AMD® Opteron® processor, ARM®. It is a processor etc. Although FIG. 1 shows one processor, the machine 105 includes any number of processors, each of which is a single-core or multi-core processor. Memory 115, a flash memory, SRAM (Static Random Access Memory), PRAM (Persistent Random Access Memory), FeRAM (Ferroelectric Random Access Memory), or MRAM (Magnetoresistive Random Access Memory) such as NVRAM (Non-Volatile Random Access Memory ) And any other variety of memory. The memory 115 may be any desired combination of different memory types. The memory 115 is controlled by a memory controller 125 that is part of the machine 105.

The storage device 120 is any variety of storage devices. The storage device 120 is controlled by the device driver 130 arranged in the memory 115.

FIG. 2 is a diagram showing further details of the machine 105 shown in FIG. Referring to FIG. 2, in general, a machine 105 comprises one or more processors 110 including a memory controller 125 and a clock 205 used to coordinate the operation of the components of the machine 105. The processor 110 is connected to a memory 115 including, for example, a RAM (random access memory), a ROM (read-only memory), or another state-holding medium. The processor 110 is coupled with a storage device 120 and a network connector 210 such as an Ethernet connector or a wireless connector. The processor 110 is connected, among other components, to the bus 215 to which the input / output interface ports managed using the user interface 220 and the input / output engine 225 are connected.

In FIG. 1 and FIG. 2, the memory 115 is a deduplication memory. The implementation of deduplication memory differs from conventional forms of memory such as DRAM (Dynamic Random Access Memory), but these differences are not relevant to the implementation of deduplication memory. Also, whether other hardware components of the machine 105, such as the processor 110, recognize a particular implementation of the memory 115 depends on whether these components need to know the physical structure of the memory 115. Dependent. This "lack of knowledge" for a particular implementation of memory 115 extends to software components, such as application programs running on machine 105. The application program transmits read and write requests to memory 115 without knowledge of whether memory 115 includes DRAM, deduplication memory, or any other form of memory.

FIG. 3 is the US Patent Application No. 15/489371 filed on April 26, 2017 and the United States published on October 5, 2017, cited as references for all purposes herein. As described in Japanese Patent Application Publication No. 2017/0286010, the use of conventional hash tables for deduplication memory in machine 105 of FIG. 1 is shown. As shown in FIG. 3, the memory 115 includes a hash table 305, a conversion table 310, a signature table 315, and an overflow area 320. The hash table 305 is configured to include a 2 ^m row or bucket and a 2 ⁿ way or column. The hash table 305 is used to store user data, and each user data is stored in a particular way within a particular hash bucket. Figure 3 shows that the hash table 305 is about 1/3 of the total memory, but in reality the hash table 305 is of arbitrary size and is often as large as possible to fit within the available memory. (To maximize deduplication memory). The overflow area 320 indicates a portion of memory 115 that is not used as deduplication memory (there is more memory than is used by hash table 305, but sufficient to double the number of ways in hash table 305. Because there is no additional memory).

FIG. 4 is a diagram showing the use of an extensible hash table according to an embodiment of the present invention. In contrast to FIG. 3, FIG. 4 includes a large hash table 305, a conversion table 310, a signature table 315, an overflow area 320, and a small hash table 405. The small hash table 405, like the large hash table 305, contains 2 ^m buckets. However, the small hash table 405 contains 2 n'way, where n'is less than ⁿ . In some embodiments of the invention, n'varies and the small hash table 405 grows dynamically over time. So, for example, n'starts from 0 or 1 (depending on the implementation) and increases by 1 when the small hash table 405 is fully filled and no new entries are placed in the bucket. For example, the small hash table 405 shrinks dynamically when the memory 115 performs relatively little deduplication. In another embodiment of the invention, the small hash table 405 is set statically (large enough that it may be in memory).

For a row and column value given to one of the large hash table 305 or the small hash table 405, the hash table contains an entry such as entry 410. Entry 410 includes data 415 and a frequency counter 420. The data 415 stores the actual data. Frequency counter 420 keeps track of the number of different references to the data. When the application is interested in using the data 415, the frequency counter 420 increases. When the application is no longer interested in data 415, the frequency counter 420 is decremented.

n'must not be greater than n. After all, if the memory contains enough space for n'to be as large as n, the large hash table 305 is created twice from the beginning and there is no need to use the small hash table 405.

The above description proposes that the large hash table 305 is statically set regardless of whether the small hash table 405 is static or dynamic. In some embodiments of the invention the large hash table 305 is statically set, whereas in other embodiments of the invention the large hash table 305 does not grow any further within physical memory. ) It grows dynamically as needed. Furthermore, there is no requirement between the large hash table 305 and the small hash table 405 which of these is static or dynamic. That is, both of the tables may be static, one may be static, the other may be dynamic, or both may be dynamic.

Using both the large hash table 305 and the small hash table 405, more memory 115 is used for deduplication memory and less memory 115 is allocated to the overflow area 320. This improves the use of memory 115 and reduces the deduplication ratio required for effective deduplication.

Some examples will be helpful. Consider a situation where the total memory capacity is 274,877,906,944 bytes (about 256 GB), the large hash table 305 has 32 ( ²⁵ ) ways, and the small hash table 405 has 16 ⁽ 24) ways. .. Table 1 below shows relevant data on the use of the hash table comparing the use of the hash table 305 alone with the use of the hash table 305 and the smaller hash table 405. As is clear from Table 1, in order to achieve an effective deduplication ratio of 3.0, the primitive deduplication ratio required for hash table 305 only is 5.4. That is, about 5.4% of the data stored in memory 115 must indicate the deduplicated data in order to achieve an effective deduplication ratio of 3.0. On the other hand, if both the large hash table 305 and the small hash table 405 are used together, only 3.9 primitive deduplication ratios are needed to achieve an effective deduplication ratio of 3.0. .. This has been significantly improved.

When both the large hash table 305 and the small hash table 405 are used, the reason why a lower primitive deduplication ratio is required is that more memory 115 is used as the deduplication memory. That is, the overflow region 320 is smaller. Since less memory 115 is used for the overflow area 320, a lower primitive deduplication ratio is required and memory 115 is generally used more efficiently.

As a second example, consider the same hash table in the same physical memory. In this example, we consider an effective deduplication ratio that assumes a constant primitive deduplication ratio. Table 2 shows such a situation. As shown in Table 2, the effective deduplication ratio when the hash table 305 is used alone is higher than the effective deduplication ratio when the large hash table 305 and the small hash table 405 are used together. Is also low.

The embodiments described above show one large hash table 305 and one small hash table 405. However, there is no reason to use only one small hash table 405. FIG. 5 is a diagram illustrating the use of an extensible hash table according to another embodiment of the invention. In this embodiment, a large number of small hash tables are supported at the expense of return on investment. For example, in FIG. 5, memory 115 is shown to include a large hash table 305, a small hash table 405, and a small hash table 505. The small hash table 505 is identical in form and function to the small hash table 405, except that the hash table 505 includes an N "way, where (N'is a power of 2 less than N). Ni) N "is a power of 2 less than N'.

The translation table 310 is responsible for mapping the logical address to the address where the desired user data is stored. Specifically, the conversion table 310 stores rows and columns (ie, buckets and ways) in a hash table containing user data (if the user data is stored in one of the hash tables). Or store the physical address in the overflow area 320 (if the user data is not stored in one of the hash tables). FIG. 6 illustrates this process. That is, FIG. 6 is a diagram showing the use of the conversion table of FIG. 4 to map logical addresses to various memory destinations.

In FIG. 6, the translation table 310 receives a logical address 605 from the host computer (the logical address 605 is an application, an operating system, or any other data that needs to be accessed from the memory 115 of FIG. 1). Ultimately obtained from software or hardware). The logical address 605 is a part of a data request which is one of a read request or a write request. The logical address 605 is considered to include two components: a translation table index and a granularity. The conversion table index indicates the specific page (or cache line) where the requested data is found. The granularity indicates a specific byte of the data to be searched. Specifically, the translation table index is generated by masking the least significant bit from the logical address 605. How many bits are masked to generate a translation table index depends on the size of the translation table index (this depends on the size of memory 115 in FIG. 1 and the size of the cache line used by the computer system. Dependent).

The conversion table index is then used as an index in the conversion table from which the PLID (Physical Line Identifier) 610 is read. The PLEID 610 takes different forms depending on where the user data is actually stored. However, in all cases, the PLEID 610 includes a region identifier 615 and a physical address 620.

If the user data is stored in the large hash table 305 of FIG. 4, the PLEID 610 looks like entry 625. At entry 625, the region identifier contains one bit, indicating that the user data was stored in the large hash table 305 of FIG. The physical address then includes m bits for the low index (identifying the hash bucket) and n bits for the column index (identifying the way). Since the m bits are sufficient to select from within the 2 ^m hash bucket and the n bits are sufficient to select from within the 2 ⁿ way, the unique user data is identified in the large hash table 305. To.

If the user data is stored in the small hash table 405 of FIG. 4, the PLEID 610 looks like entry 630. At entry 630, the region identifier contains 2 bits. That is, it includes a first bit indicating that the user data is not stored in the large hash table 305 of FIG. 4 and a second bit indicating that the user data is stored in the small hash table 405 of FIG. .. The physical address then includes an m-bit for the low index (identifying the hash bucket) and an n'bit for the column index (identifying the way). Since n'is always less than n, entry 630 does not require any more bits than entry 625, even though 2 bits are used to identify the area where user data is stored.

If the user data is stored in the overflow area 320 of FIG. 4, the PLEID 610 looks like entry 635. At entry 635, the region identifier again contains 2 bits. That is, it includes a first bit indicating that the user data is not stored in the large hash table 305 of FIG. 4, and a second bit indicating that the user data is stored in the overflow area 320 of FIG. The physical address contains raw and column indexes (such as entries 625, 630) or is formatted in any desired format using any other desired format.

Although entries 625, 630, and 635 show how to distinguish between entries in the large hash table 305 of FIG. 4, the small hash table 405 of FIG. 4, and the overflow region 320 of FIG. 4, the present invention is practiced. The morphology supports the other morphology used. For example, in an embodiment comprising a large number of small hash tables 405 and 505 shown in FIG. 5, the region identifier 615 includes one bit for distinguishing between the large hash table 305 of FIG. 5 and the other region. , The small hash table of FIG. 5 (405, 505), and 2 bits for selection from the overflow area 320 of FIG. Alternatively, the region identifier 615 may always use 2 bits to select up to 4 different regions, or may always use 3 bits to select up to 8 different regions. This approach has the advantage that it does not require a diverse number of bits to select the region, but it does require a diverse number of bits to store the entire PLEID 610 and some of the bits for the unused region 615. The combination remains.

FIG. 6 describes the conversion table 310 as a table, with entries keyed off many bits from the logical address, but the conversion table 310 is embodied using other techniques, such as hash functions. When using a hash function, a specific logical address (or the associated high-order bit of the logical address) may be the target of the hash function. The result of the hash function is used to determine the location of the user data (large hash table 305 in FIG. 4, small hash table 405 in FIG. 4, or overflow area 320 in FIG. 4) and physical address.

Returning to FIG. 4, one problem when using the deduplication memory is to check whether the user data is actually stored somewhere in the memory 115. For example, different applications request access to the same data, but use different logical addresses (because neither application recognizes the interest of other applications or user data). The signature table 315 determines if the given user data is a duplicate of some other user data that already exists in memory 115 and stores an unnecessary copy of the user data. Used to prevent.

When the user data is newly stored in memory 115, the hash function is applied to the user data to generate a signature. This hash function is the same as or different from the hash function used to determine where the user data is actually stored in memory. Unlike the hash function used to determine where the user data is stored, the hash function used to generate the signature is not the logical address of the user data, but the hash of the user data itself. To carry out. The signature table 315 is then searched to see if the signature exists.

Signatures are usually shorter (ie, less bits) in length than the user data itself. Therefore, it is possible for different user data to generate the same signature. In other words, if a signature match is found in the signature table 315, that match does not automatically mean that the user data is already stored in memory 115. User data is compared with the identified data in memory 115 to determine if the user data was actually stored in memory 115. If the exact comparison shows a match, the user data is already stored in memory 115. In that case, the conversion table 310 is set so that the PLL 610 in FIG. 6 indicates the position where the user data is stored. Keep in mind that the opposite proposition is true. If no signature match is found in the signature table 315, the user data is not yet stored in memory 115 (because the same data does not use the same hash function to generate different signatures). In that case, the new entry is added to the signature table 315, which is remapped to the logical address.

The signature table 315 is generally used only for user data stored in the large hash table 305 and the small hash table 405. That is, the signature table 315 is not used for the data stored in the overflow area 320. The reason is simple. Since the overflow area 320 is not a target of deduplication, duplicate data may be stored there.

7A and 7B are flowcharts showing an example of a procedure for using the expandable hash table of FIG. 4 in the deduplication memory according to one embodiment of the present invention. At step 705 of FIG. 7A, memory 115 of FIG. 4 receives the logical address 605 of FIG. 6 from machine 105 of FIG. 1 as part of a data request (machine 105, operating system, or some other). From the application program running on the component). At step 710, the memory 115 of FIG. 4 determines the PLEID 610 of FIG. 6 corresponding to the logical address 605 of FIG. In this case, since different approaches are used to determine the PLLD 610 of FIG. 6, how the PLLD 610 of FIG. 6 is determined depends on whether the data is read or written. A flowchart of how the PLLD 610 of FIG. 6 is discriminated against a read and write request is shown with reference to FIGS. 8 and 9A-9C below, respectively.

When the PLLD 610 of FIG. 6 is determined, in step 715 the memory 115 of FIG. 4 uses the area identifier of the PLL ID 610 of FIG. 6 to determine where the data is stored. When the data is stored in the overflow area 320 of FIG. 4, in step 720 the user data (depending on the type of request issued, either read the data from the overflow area 320 of FIG. 4 or read the data from the overflow area 320 of FIG. 4). After the process of writing data to) is completed, the data is accessed from the PLL 610 in FIG. 6 using the physical address. Otherwise, at step 725, memory 115 in FIG. 4 has a low index and column index (ie, hash bucket and way) in one of the large hash table 305 of FIG. 4 or the small hash table 405 of FIG. Determination (data access is the same except where the hash table is stored in memory 115 of FIG. 4).

At step 730 (FIG. 7B), memory 115 of FIG. 4 determines whether the requested data has been "discovered" at a particular low index and column index. For example, when data is written to one of the hash tables, it happens that the raw and column indexes identify where the data is already (less likely, but possible). In that case, the user data is written to a nearby location (eg, within some predetermined delta of the column index, other ways in the same hash bucket), so the data is in that location instead of being read. Need to be searched for. What is considered "close" in the hash table is further discussed below with reference to FIGS. 9A-9C. In the case of a write request, the data "discovered" means that there is an available entry in the hash table to which the data is written.

In the situation where the logical address 605 of FIG. 6 is a read request portion, the memory 115 of FIG. 4 has no way of determining that the data of a specific low index and column index is not the requested data. In such a situation, step 730 generally always returns a "Yes" result to access the data and automatically proceeds to step 735.

If the data is found at a particular low index and column index, or if the particular low index and column index can store the data for a write request, then in step 735 the memory 115 of FIG. 4 accesses the data and then. The process is completed. If not, data will be written, but if the low index and column index identify where the data is already stored, then in step 740 memory 115 in FIG. 4 is for the position where the user data is written. Search for nearby entries. At step 745, if there is no location near where user data is written, memory 115 of FIG. 4 reports an error at step 750. Or, in particular, if the data is to be written to one of the hash tables but cannot be written for some reason, the PLEID 610 in FIG. 6 should point to the overflow area 320 in FIG. 4 instead of the hash table. (In this case, the process proceeds to step 720 in FIG. 7A). Otherwise, the data is "discovered" in the vicinity, user data is accessed from the hash table at step 735, and processing is complete.

FIG. 8 is a flowchart showing an example of a procedure for determining a PLID (Physical Line Identifier) for a logical address in a memory read request according to an embodiment of the present invention. For read requests, the process is simple. At step 805, the logical address is used to access the translation table 310 of FIG. The translation table 310 determines the physical address used to access the data from the appropriate area in memory 115 of FIG. 4 (one of the hash tables includes a low index and a column index). PLID 610 is provided.

9A-9C are flowcharts showing an example of a procedure for determining a PLID for a logical address in a memory write request according to an embodiment of the present invention. At step 905 of FIG. 9A, memory 115 of FIG. 4 generates a signature of the data to be written. At step 910, the signature table 315 of FIG. 4 confirms whether or not a signature exists. When a signature match is found, in step 915 the memory 115 of FIG. 4 is the data whose data is stored in a hash table (one of the large hash table 305 of FIG. 4 or the small hash table of FIG. 4). Check if it matches with. If not, at step 920, memory 115 in FIG. 4 checks to see if signature table 315 in FIG. 4 has space for new entries.

When the data matches the data stored in the hash table in step 915, in step 925 (FIG. 9B), the memory 115 in FIG. 4 overflows the frequency counter 420 in FIG. 4 if the frequency counter 420 in FIG. 4 increases. Check if it is done (occurs when the frequency counter 420 in FIG. 4 has already reached the maximum value). In that case, or if the signature table is in step 920 of FIG. 9A and has no space for a new entry, user data is written to the overflow area 320 of FIG. 4, and in step 930 the memory 115 is relative to the overflow area 320 of FIG. Generate the PLL 610 of FIG. Alternatively, if the signature is found in the signature table 315 of FIG. 4, the data matches the entry in the hash table, the frequency counter 420 of FIG. 4 does not overflow, and the memory 115 of FIG. 4 at step 935 is of FIG. The frequency counter 420 is increased, and the process ends. When the frequency counter 420 of FIG. 4 is increased without overflowing, the data does not need to be further written to the memory 115 of FIG. 4 (already stored in the hash table), and therefore the processing of FIG. 7A is also completed. do.

If the signature is found in the signature table 315 of FIG. 4, but the data do not match at stage 915, then the particular combination of hash buckets and ways that the logical address normally maps to is already in use. This situation is referred to as a "hash collision". When hash collisions occur, there are several ways to deal with them. As shown in step 940, one possibility is to look for a new available location in the hash table and update the PLLD 610 in FIG. 6 to point to the new location. The second possibility is to advance control to step 930 and write data to the overflow area 320 of FIG. 4 instead of the hash table, thereby updating the PLID of FIG. 6 again. The third possibility, as shown in step 945, is to leave the PLLD 610 of FIG. 6 unchanged and to the memory 115 of FIG. 4 to know that the data is actually stored somewhere else. That is. There are existing solutions to hash collisions, for example open addressing, where the data is not stored in the exact location identified, but is free after the hash collision occurs. Stored somewhere in the location. When using the open address method, the data is stored at any address after the position identified by PLID610 in FIG. 6, or the data is stored within a fixed, predetermined number of positions. (A fixed, predetermined number is set to any desired value). In another embodiment of the invention, the data is stored again in a number of fixed and predetermined numbers of positions prior to the positions identified by PLLD610 in FIG. Regardless of which approach is used, the memory 115 of FIG. 4 will generate the PLLD 610 of FIG. 6 and use this approach to access the data from the location specified by the PLL 610 of FIG. Embodiments of the invention also support other solutions to hash collisions.

If no signature is found in the signature table 315 of FIG. 4 and the signature table 315 of FIG. 4 has space for a new entry, then at stage 950 (FIG. 9C), the signature is added to the signature table 315 of FIG. At 955, memory 115 produces PLID 610 of FIG. 6 for one of the large hash table 305 of FIG. 4 and the small hash table 405 of FIG. 4, depending on where the available entry is found.

FIG. 10 is a flowchart showing an example of a procedure for determining whether or not to increase the size of a small hash table according to an embodiment of the present invention. At step 1005 in FIG. 10, it is confirmed that the small hash table 405 in FIG. 4 is approaching the capacity. Also, the memory 115 of FIG. 4 confirms whether the overflow area 320 of FIG. 4 has sufficient storage space available for use in the small hash table 405 of FIG. Since the small hash table 405 of FIG. 4 is doubled in size, the overflow area 320 of FIG. 4 needs to have at least enough space for the small hash table of FIG. 4 to already use. It takes as much space as possible because some data may already be stored in the overflow area 320 of FIG. 4 and needs to be left in the overflow area 320 of FIG. If the small hash table 405 of FIG. 4 has sufficient space, or the overflow area 320 of FIG. 4 does not have sufficient storage space to support the increasing small hash table 405 of FIG. 4, the process ends. .. If not (and assuming that the value n'for the small hash table 405 in FIG. 4 is at least 2 less than the value n for the large hash table 305 in FIG. 4), in step 1010 the size of the overflow region 320 in FIG. 4 is increased. In return for the reduction, the small hash table 405 in FIG. 4 is increased in size. Then, at step 1015, the small hash table 405 of FIG. 4 doubles the number of ways (by increasing the value n') to use the newly added memory and the process ends.

7A-10 disclose some embodiments of the present invention. However, those with ordinary knowledge in the art may also implement other embodiments of the invention by rearranging the order of the steps, omitting the steps, or including concatenation not shown in the drawings. It is possible. All such modifications of the flow chart are considered embodiments of the invention, whether explicitly stated or not.

The following description provides a brief and general description of a suitable machine that embodies certain aspects of the technical idea of the present invention. Machines, at least in part, are existing such as instructions received from other machines, interaction with virtual reality (VR), biometric feedback, or other input signals, as well as keyboards, mice, etc. It is controlled by the input from the input device of. As used herein, the term "machine" broadly includes one machine, a virtual machine, a system of communicably coupled machines, or devices that work together. Examples of machines include personal computers, workstations, servers, portable computers, handheld devices, telephones, tablets, as well as transportation equipment such as personal or public transportation such as automobiles, trains, taxis, etc. Includes computing devices such as.

Machines include programmable or non-programmable logic devices or arrays, embedded controllers such as application specific integrated circuits (ASICs), embedded computers, smart cards, and the like. The machine utilizes one or more connections to one or more remote machines, for example through a network interface, modem, or other communication combiner. The machines are interconnected by physical and / or logical network means such as intranet®, internet, LAN (local area network), WAN (wide area network) and the like. We have a variety of wired and / or wireless short-range or wireless network communication including wireless (RF), satellite, microwave, 802.11 standard, Bluetooth®, optical, infrared, cable, laser, etc. Understand that long range carriers and protocols can be used.

Embodiments of the invention, when accessed by a machine, perform a function, procedure, data structure, application that results in the machine performing business or defining an abstract data type or low level hardware context. Refer to related data including programs, etc., or be explained together with related data. Related data includes, for example, volatile and / or non-volatile memory (eg, RAM, ROM, etc.), or other storage devices, and hard drives, floppy disks, optical storage devices, tapes, flash memories, memory sticks, digital video. It is stored in a related storage medium including a disk, biological storage device, and the like. Related data is transmitted in the form of packets, serial data, parallel data, propagated signals, etc. through transmission environments including physical and / or logical networks and is used in compressed or encrypted formats. Relevant data is used in a distributed environment and is stored locally and / or remotely to machine access.

Embodiments of the invention include a medium that can be decoded by a non-temporary machine of the type including instructions that can be executed by one or more processors, where the instructions constitute the concept of the invention described herein. Contains instructions to carry out the element.

Although the embodiments of the present invention have been described above with reference to the drawings, the present invention is modified without departing from the technical scope of the present invention and combined with other desirable methods. Also, although the description herein has focused on a particular embodiment, other configurations may be considered. In particular, expressions such as "according to embodiments of the present invention" are used herein, such clauses mean to generally refer to the possibilities of embodiments, and particular embodiments. It does not limit the present invention. As used herein, these terms refer to the same or different embodiments that can be combined with other embodiments.

The present invention is not limited to the above-described embodiment, and various modifications can be made to such an embodiment. Therefore, all such modifications are within the scope of the present invention.

The embodiments of the present invention are extended without limitation to the following description.

Explanation 1. The memory system according to the embodiment of the present invention includes a memory for storing data and a memory system.
A large hash table stored in the memory, containing a predetermined number of buckets and a first number of ways, and containing a first portion of the memory containing a first power of two, a first number of bytes.
A small hash table stored in the memory, containing a predetermined number of buckets and a second number of ways, and a second portion of the memory containing a second power of 2 bytes.
An overflow area stored in the memory and including a third portion of the memory,
A memory system including a conversion table for mapping a logical address to a PLID (Physical Line Identity) including an area identifier and a physical address is provided.

Explanation 2. An embodiment of the invention includes a memory system according to Description 1, wherein the area identifier includes a first bit indicating that the PLID identifies data in the large hash table.

Explanation 3. An embodiment of the present invention includes a memory system according to Description 2, wherein the physical address includes a low index and a column index.

Explanation 4. Embodiments of the present invention include a memory system according to Description 2.
The first bit indicates that the PLID does not identify the data in the large hash table.
The region identifier includes a second bit indicating whether the PLID is in the small hash table or in the overflow region.

Explanation 5. Embodiments of the present invention include a memory system according to Description 4.
The second bit indicates that the PLID classifies the data in the small hash table by job.
The physical address includes a low index and a column index.

Explanation 6. An embodiment of the present invention includes a memory system according to Description 2, wherein the small hash table dynamically grows.

Explanation 7. An embodiment of the present invention includes a memory system according to Description 6, wherein the large hash table dynamically grows.

Explanation 8. Embodiments of the invention include the memory system according to Description 1, where the first effective minimum deduplication ratio for the memory system is the second effective minimum for the large hash table without the small hash table. Is lower than the deduplication ratio of.

Explanation 9. Embodiments of the present invention include a memory system according to Description 1.
Further including a signature table stored in the memory and storing a signature of the large hash table and a plurality of data stored in the small hash table.
The signature table prevents a large amount of data having a common signature from being stored in the row of the large hash table or the small hash table.

Explanation 10. The method according to the embodiment of the present invention
The stage of receiving a logical address from the processor and
A step of mapping the logical address to a PLID (Physical Line Identity) including a region identifier and a physical address using a translation table, and a step of mapping.
The step of determining whether or not the physical address exists in the large hash table, the small hash table, or the overflow area in the memory by using the area identifier, and
It has a stage of accessing data in the memory by using the physical address.

Explanation 11. An embodiment of the present invention includes a method according to the description 10, and uses the area identifier to determine whether or not the physical address is present in a large hash table, a small hash table, or an overflow area in a memory. Includes a step of determining that the physical address is in the large hash table if the first bit of the region identifier is not set.

Explanation 12. An embodiment of the present invention includes a method according to the description 10, and uses the area identifier to determine whether or not the physical address is present in a large hash table, a small hash table, or an overflow area in a memory. Includes the step of determining that the physical address is in the small hash table when the first bit of the area identifier is set and the second bit of the area identifier is not set.

Explanation 13. An embodiment of the present invention includes a method according to the description 10, and uses the area identifier to determine whether or not the physical address is present in a large hash table, a small hash table, or an overflow area in a memory. Includes a step of determining that the physical address is in the overflow region when the first bit of the region identifier is set and the second bit of the region identifier is set.

Explanation 14. The embodiment of the present invention includes the method according to the description 10, and the step of accessing the data in the memory by using the physical address is
The stage of determining the low index and column index from the physical address, and
The low index and the column index are used to access one of the data in the large hash table and the small hash table.

Explanation 15. Embodiments of the present invention include the method according to Description 14, and the step of accessing one data in the large hash table and the small hash table using the low index and the column index is the small hash. If the data is not found in the low index and the column index of the table, it comprises searching for entries near the small hash table.

Explanation 16. The embodiment of the present invention includes the method according to the description 10, and the step of accessing the data in the memory by using the physical address is the step of accessing the data in the overflow area by using the physical address. include.

Explanation 17. Embodiments of the present invention include a method according to Description 10.
At the stage of determining that the small hash table is approaching the capacity,
It further comprises a step of reducing the size of the overflow area while increasing the size of the small hash table.

Explanation 18. Embodiments of the present invention include a method according to Description 17.
The step of increasing the size of the small hash table is
At the stage of doubling the size of the small hash table,
Includes a step of reducing the size of the overflow region.

Explanation 19. An embodiment of the present invention comprises the method according to Description 17, wherein the step of increasing the size of the small hash table comprises increasing the number of ways corresponding to the columns of the small hash table.

Explanation 20. Embodiments of the present invention include a method according to Description 10.
The step of accessing the data in the memory using the physical address includes the step of writing the data to the memory.
The step of using a translation table to map the logical address to a PLID containing a region identifier and a physical address is to use the translation table to write the data to the large hash table, the small hash table, and the small hash table. It comprises the step of selecting one of the overflow areas.

Explanation 21. An embodiment of the present invention includes the method according to the description 20, and uses the area identifier to determine whether or not the physical address is in a large hash table, a small hash table, or an overflow area in the memory. teeth,
At the stage of applying a hash function to the data to produce a signature,
At the stage of checking the signature table for the signature,
If the signature is in the signature table, it includes writing the data to the overflow area.

Explanation 22. An embodiment of the invention comprises the method according to Description 21, wherein the step of checking the signature table for the signature comprises checking the signature table for the signature at the row of the physical address.

Explanation 23. The computer-readable recording medium according to the embodiment of the present invention records a program for causing the computer to execute the following method, and ...
The method is
The stage of receiving a logical address from the processor and
A step of mapping the logical address to a PLID (Physical Line Identity) including a region identifier and a physical address using a translation table, and a step of mapping.
The step of determining whether or not the physical address exists in the large hash table, the small hash table, or the overflow area in the memory by using the area identifier, and
It has a step of accessing data in the memory using the physical address.

Explanation 24. An embodiment of the present invention includes a recording medium according to the description 23, and uses the area identifier to determine whether or not the physical address is in a large hash table, a small hash table, or an overflow area in a memory. The step includes determining that the physical address is in the large hash table if the first bit of the region identifier is not set.

Explanation 25. An embodiment of the present invention includes a recording medium according to the description 23, and uses the area identifier to determine whether or not the physical address is in a large hash table, a small hash table, or an overflow area in a memory. The step includes the step of determining that the physical address is in the small hash table when the first bit of the area identifier is set and the second bit of the area identifier is not set.

Explanation 26. An embodiment of the present invention includes a recording medium according to the description 23, and uses the area identifier to determine whether or not the physical address is in a large hash table, a small hash table, or an overflow area in a memory. The step includes a step of determining that the physical address is in the overflow region when the first bit of the region identifier is set and the second bit of the region identifier is set.

Explanation 27. The embodiment of the present invention includes a recording medium according to the description 23, and the step of accessing the data in the memory by using the physical address is
The stage of determining the low index and column index from the physical address, and
The low index and the column index are used to access one of the data in the large hash table and the small hash table.

Explanation 28. The embodiment of the present invention includes a recording medium according to the description 27, and the step of accessing one data in the large hash table and the small hash table by using the low index and the column index is small. If the data is not found in the low index and the column index of the hash table, it includes a step of searching for entries in the vicinity of the small hash table.

Explanation 29. The embodiment of the present invention includes a recording medium according to the description 23, and the step of accessing the data in the memory by using the physical address is the step of accessing the data in the overflow area by using the physical address. including.

Explanation 30. An embodiment of the present invention comprises a recording medium according to Description 23, wherein the method is:
At the stage of determining that the small hash table is approaching the capacity,
It further comprises a step of reducing the size of the overflow area while increasing the size of the small hash table.

Explanation 31. An embodiment of the present invention includes a recording medium according to the description 30.
The step of increasing the size of the small hash table is
At the stage of doubling the size of the small hash table,
Includes a step of reducing the size of the overflow region.

Explanation 32. An embodiment of the present invention includes a recording medium according to Description 30, and the step of increasing the size of the small hash table comprises increasing the number of ways corresponding to the columns of the small hash table.

Explanation 33. An embodiment of the present invention includes a recording medium according to the description 23, and comprises a recording medium.
The step of accessing the data in the memory using the physical address includes the step of writing the data to the memory.
The step of using a translation table to map the logical address to a PLID containing a region identifier and a physical address is to use the translation table to write the data to the large hash table, the small hash table, and the small hash table. It comprises the step of selecting one of the overflow areas.

Explanation 34. An embodiment of the present invention includes a recording medium according to the description 33, and uses the area identifier to determine whether or not the physical address is in a large hash table, a small hash table, or an overflow area in a memory. The stage is
At the stage of applying a hash function to the data to produce a signature,
At the stage of checking the signature table for the signature,
If the signature is in the signature table, it includes writing the data to the overflow area.

Explanation 35. An embodiment of the present invention includes a recording medium according to the description 34, and a step of checking the signature table for the signature includes a step of checking the signature table for the signature at the row of the physical address.

105 Machine 110 Processor 115 Memory 120 Storage Device 125 Memory Controller 130 Device Driver 205 Clock 210 Network Connector 215 Bus 220 User Interface 225 Input / Output Engine 305 (Large) Hash Table 310 Conversion Table 315 Signature Table 320 Overflow Area 405, 505 Small Hash table 410, 625, 630, 635 Entry 415 Data 420 Frequency counter 605 Logical address 610 PLID (Physical Line Processor)
615 Area identifier 620 Physical address

Claims (16)

Memory to store data and
A large hash table stored in the memory, containing a predetermined number of buckets and a first number of ways, and containing a first portion of the memory containing a first power of two, a first number of bytes.
A small hash table stored in the memory, containing a predetermined number of buckets and a second number of ways, and a second portion of the memory containing a second power of 2 bytes.
An overflow area stored in the memory and including a third portion of the memory,
A memory system comprising: a conversion table for mapping a logical address to a PLID (Physical Line Identity) including an area identifier and a physical address. The memory system according to claim 1, wherein the area identifier includes a first bit indicating that the PLID identifies data in the large hash table. The memory system according to claim 2, wherein the physical address includes a low index and a column index. The first bit indicates that the PLID does not identify the data in the large hash table.
The memory system according to claim 2, wherein the area identifier includes a second bit indicating whether the PLID is in the small hash table or in the overflow area. The memory system according to claim 1, wherein the small hash table dynamically grows. Claim 1 characterized in that the first effective minimum deduplication ratio for the memory system is less than the second effective minimum deduplication ratio for the large hash table without the small hash table. The memory system described in. Further including a signature table stored in the memory and storing a signature of the large hash table and a plurality of data stored in the small hash table.
The memory system according to claim 1, wherein the signature table prevents a large amount of data having a common signature from being stored in the row of the large hash table or the small hash table. The stage of receiving a logical address from the processor and
A step of mapping the logical address to a PLID (Physical Line Identity) including a region identifier and a physical address using a translation table, and a step of mapping.
The step of determining whether or not the physical address exists in the large hash table, the small hash table, or the overflow area in the memory by using the area identifier, and
A method characterized by having a step of accessing data in the memory using the physical address. When the first bit of the area identifier is not set, the step of determining whether or not the physical address exists in the large hash table, the small hash table, or the overflow area in the memory by using the area identifier is described. The method of claim 8, wherein the method comprises the step of determining that the physical address is in the large hash table. In the step of determining whether or not the physical address exists in the large hash table, the small hash table, or the overflow area in the memory by using the area identifier, the first bit of the area identifier is set and the area is set. The method according to claim 8, wherein when the second bit of the identifier is not set, the step of determining that the physical address is in the small hash table is included. In the step of determining whether or not the physical address exists in the large hash table, the small hash table, or the overflow area in the memory by using the area identifier, the first bit of the area identifier is set and the area is set. The method according to claim 8, wherein when the second bit of the identifier is set, the step of determining that the physical address is in the overflow area is included. The stage of accessing the data in the memory using the physical address is
The stage of determining the low index and column index from the physical address, and
The method according to claim 8, wherein the method comprising the step of accessing one data in the large hash table and the small hash table by using the low index and the column index. At the stage of accessing one data in the large hash table and the small hash table by using the low index and the column index, the data is not found in the low index and the column index of the small hash table. The method of claim 12, wherein the method comprises searching for entries near the small hash table. At the stage of determining that the small hash table is approaching the capacity,
8. The method of claim 8, further comprising a step of reducing the size of the overflow area while increasing the size of the small hash table. 14. The method of claim 14, wherein the step of increasing the size of the small hash table comprises increasing the number of ways corresponding to the columns of the small hash table. A computer-readable recording medium on which a program for causing a computer to execute the method according to any one of claims 8 to 12 is recorded.

Images