DE102013208930B4 - Grouping of entries in a deduplication index - Google Patents

Abstract

Computer program product for performing a deduplication for a memory, the computer program product having a computer-readable storage medium with a computer-readable program code embodied therein, with the steps of: - managing an index of data block signatures calculated from data blocks of data in data objects in the memory, each index entry contains at least one of the data block signatures from the plurality of index entries and a reference to the data block from which the signature was calculated; - selecting entries in the index which are to be combined by determining entries from the plurality of entries in the index the basis of a criterion of the data blocks of data referenced in the entries, the criterion ◯ an age of the data blocks in the memory, the age being a period of time since the data block was inserted into the memory or after it was updated; ◯ and / or a Hau ability of input / output (I / O) accesses to the data blocks in the memory; ◯ and / or a frequency with which the data blocks appear in the memory in data objects that have been inserted into the memory; ◯ and / or a Area of the memory in which the data blocks are stored; ◯ and / or a data type in the data blocks; - Carrying out a summarization operation on the data block signatures in the selected entries to generate a summarized signature; - Insertion of an entry in the index, the summarized signature and a reference to the data blocks in memory referenced in the summarized selected entries; - removing from the index the selected entries that were used to generate the summarized signature, the resulting summarized entries and from the summarization operation unaffected entries refer to all data blocks in memory that are in memory it contains managed data objects; and using the index of the signatures in deduplication operations when inserting data objects into memory.

Description

The present invention relates to a computer program product, a system and a method for combining entries in a deduplication index.

Data deduplication is a data reduction technique used to remove redundant data to improve storage utilization. Deduplication reduces the amount of storage required by storing only one copy of a particular unit of data, also known as a data block. In the case of disk storage systems such as a storage management server or a virtual tape library (VTL), deduplication technology can be used to identify redundant data blocks and avoid duplicate entries as a result of the redundant storage of such data blocks. Data deduplication in storage reduces the amount of storage required for a given set of files and is most effective in applications where copies of very similar or even identical data are stored on a single disk, which is often the case. With data backups that are carried out continuously to protect against data loss, most of the data in a particular data backup has not changed compared to the previous data backup and can often give rise to deduplication in order to bypass the redundant storage of data.

Data deduplication can take place at the file or block level. File deduplication removes duplicate files. Block deduplication searches a file and saves the unique occurrence of each block. A block deduplication system divides a file into a sequence of data blocks. The deduplication system determines whether some of the data blocks are already stored and then only saves these non-redundant data blocks. The redundancy check can be performed with the data blocks that are either currently being saved in the file or are already saved in the system.

An object can be divided into data blocks using an electronic fingerprinting technique such as the Rabin-Karp fingerprinting technique. Redundant data blocks are recognized by applying a hash algorithm such as MD5 (Message Digest Algorithm 5) or SHA-1 (Secure Hash Algorithm 1) to each data block in order to generate a hash value for the data blocks and then include these hash values to compare the hash values of data blocks already stored in the system. The hash values for stored data blocks are usually managed in an index (deduplication index). A data block can be uniquely identified by a hash value and / or a fingerprint. If an entry is found for a hash value, a redundant data block is recognized and this data block in the data object or object can be replaced by a pointer to the corresponding data block maintained in the memory. The size of the deduplication index can be limited by the amount of storage available for the index. This can limit the number of memory-referencing hash values that can be maintained in the deduplication index. Setting up a smaller index can ensure that the index fits into the allocated storage volume.

US 2011/099351 A1 discloses the routing of data for improved deduplication in a memory. New or changed data is loaded into the memory and a hash content value and a hash similarity value are calculated for the data.

The object of the invention is to further improve the performance of deduplication operations.

The object is achieved according to the invention by the computer program product according to claim 1, the system according to claim 6 and the method according to claim 7. Preferred embodiments of the invention are the subject matter of the respective subclaims.

• 1 veranschaulicht eine Datenverarbeitungsumgebung, in der Ausführungsformen realisiert werden.
• 2 veranschaulicht eine Ausführungsform von Objektdaten.
• 3 veranschaulicht eine Ausführungsform eines Eintrags des Deduplizierungs-Indexes.
• 4 veranschaulicht ein Beispiel eines Deduplizierungs-Indexes mit zusammengefassten Einträgen.
• 5 veranschaulicht eine Ausführungsform von Operationen zum Einfügen eines Datenblocks in den Deduplizierungs-Index.
• 6 veranschaulicht ein Beispiel von Operationen zum Zusammenfassen von Indexeinträgen in dem Deduplizierungs-Index.
• 7a und 7b veranschaulichen eine Ausführungsform von Operationen zum Verarbeiten eines empfangenen Datenobjekts zum Deduplizieren durch Berechnen zusammengefasster Signaturen.
• 8 veranschaulicht eine Ausführungsform einer Zuordnung von Datenblöcken in einem empfangenen Datenobjekt mit einer zusammengefassten oder Datenblocksignatur, die von einem oder mehreren Datenblöcken in dem empfangenen Datenobjekt berechnet wurden.
• 9 veranschaulicht eine Ausführungsart eines Computerknotens in der Datenverarbeitungsum gebung.

A computer program product, system, and method are provided for aggregating entries in a deduplication index. An index has data block signatures calculated from data blocks of data in the data objects in the memory, each index entry having at least one of the data block signatures and a reference to the data block from which the signature was calculated. Entries in the index are selected to be summarized and a summarize operation is performed on the data block signatures in the selected entries to produce a summarized signature. An entry is inserted into the index containing the pooled signature and a reference to the data blocks in memory referenced in the pooled selected entries. The signature index is used in deduplication operations when data objects are inserted into memory.

• 1 Figure 3 illustrates a computing environment in which embodiments are implemented.
• 2 illustrates one embodiment of object data.
• 3 Figure 3 illustrates one embodiment of a deduplication index entry.
• 4th illustrates an example of a deduplication index with aggregated entries.
• 5 Figure 3 illustrates one embodiment of operations to insert a block of data into the deduplication index.
• 6th Figure 11 illustrates an example of operations to merge index entries in the deduplication index.
• 7a and 7b illustrate one embodiment of operations for processing a received data object for deduplication by computing aggregated signatures.
• 8th Figure 3 illustrates an embodiment of an association of data blocks in a received data object with a summarized or data block signature calculated from one or more data blocks in the received data object.
• 9 Figure 3 illustrates one embodiment of a computing node in the computing environment.

Described embodiments provide techniques for pooling entries in a deduplication index to reduce the number of index entries by performing a pooling operation on the signatures contained in multiple entries to generate a pooled signature contained in a new pooled entry that is based on the multiple blocks of data referenced by the signatures affected by the summarize operation. If, in the described embodiments, a receiving data object is added to the memory, data block signatures are calculated from the data blocks in the received data object and combined signatures are calculated from several signatures in the received data object in order not to compare them with combined and data block signatures in the deduplication index to determine duplicate or non-matching data blocks in the received data object, ie one or more data blocks with signatures that do not match the signatures in the index. Only those data blocks that do not occur twice or only once and that are not duplicated are written to the memory in order to avoid writing redundant data.

1 Figure 11 illustrates one embodiment of a computing environment having a server 2 , the data objects over a network 6th of clients 4th receives and processes. The objects consist of data blocks 8th of data that the server 2 in a store 10 saves. The server 2 assigns a memory manager 12 for managing the storage of data objects in the memory 10 on. The storage manager 12 manages object data 14th that contains information about the objects in memory 10 exhibit. The object data 14th can use an ordered list of data blocks 8th and the address (s) of these data blocks 8th in the memory 10 provide. The one in the object data 14th objects shown, their data blocks 8th of data in the memory 10 are stored, files, database tables, multimedia content, programs, data sets or any other data arrangements that can be provided for storage. The storage manager 12 can provide storage space for active data objects that the clients 4th used in real-time operations, and / or a backup manager for backing up and archiving data objects of the client 4th exhibit.

The storage manager 12 maintains a deduplication index 16 , the information about the data blocks 8th in the memory 10 to ensure that when storing received data objects that have data blocks, only one copy of a data block 8th , ie no duplicate copies, in memory 10 is saved, although in several data objects on one data block 8th can be referenced. The storage manager 12 can also data block data 18th manage the information about the storage space of the data blocks 8th in the memory 10 exhibit. Due to limited memory resources on the server, the deduplication index 14th not one entry for each individual data block 8th manage in the memory referenced in data objects that are in the object data 14th are displayed. The data block data 16 contain information about all data blocks 8th in the memory 10 to access the data blocks 8th to be able to access.

The server 2 can data with clients 4th over a network 6th exchange, for example via a local area network (LAN), a network for the storage area (SAN), a wide area network (WAN), a cloud computing model, etc. According to further embodiments, the server 2 and the client 4th be connected to one another via a direct line or have components on the platform of a single computer system and exchange data via a bus or memory.

The memory 10 may have storage media implemented in one or more storage units known in the art, e.g. interconnected hard disk drives (configured as DASD, RAID, JBOD, etc., for example), semiconductor storage units (e.g. EEPROM (electrically erasable programmable read-only memories) , Flash memory, flash disk storage, storage-class memory (SCM), electronic storage, magnetic tape media, tape cartridges, etc.

The storage manager 12 may have a software program in a memory processed by a processor of the server 2 is performed. According to an alternative embodiment, the entire memory manager 12 or a part of it can be implemented as a hardware component, for example as an integrated special circuit, e.g. application-specific circuit (ASIC), expansion card, etc.

2 Figure 3 illustrates one embodiment of an instance of object data 50 for a data object that is in the object data 14th is included. The object data 50 for a data object contain a data object name 52 that the data object receives through the client 4th assigned, for example a unique identifier and one or more data block identifiers (Ci) 54a , 54b .... 54n of data blocks which the data objects 52 assigned to the client. The data blocks themselves can have a variable length, ie the same or different lengths, or a fixed, predetermined length. The object data 50 can provide an ordered list of data block identifiers ( C1 .... Cn), which indicates a sequence of 1 .... n in which the data blocks occur in the data object.

According to one embodiment, the data block identifier (Ci) 54a ... 54n is calculated by processing the data block data to generate a unique value for the data block, for example using the MD5 (Message-Digest Algorithm 5 ) or SHA-1 (Secure Hash Algorithm 1 ), which calculates cryptographic hash values of data blocks in a data object. According to an alternative embodiment, the data block identifier 54a , 54b ... 54n a direct reference to a memory location in the memory instead of a hash value of the data 8th exhibit.

3 illustrates one embodiment of an entry 70 in the deduplication index 16 for data blocks 8th in the memory 10 . The entry 70 can be on a single data block or a large number of data blocks 8th refer to a summarized entry. The index entry 70 contains a signature (Si) 72 that from a single block of data 8th can be calculated, for example according to the calculation of the data block identifiers 54a , 54b ... 54n or by performing a summarization operation, such as a multiplication or other operation, on previously calculated data block signatures 72 of existing index entries 70 to get a summarized data block signature. A memory reference 74 refers to the one or more consecutive data blocks 8th in the memory 10 that the signature 72 are associated. A summary level 76 indicates an extent to which the signature (Si) 72 directly from a single block of data 8th or from several signatures 72 in existing entries 70 was calculated. For example, shows a summary level 76 null a data block signature 72 directly from the data block 8th has been calculated and a memory reference 74 has on the data block 8th from which the signature was calculated. A summary level 76 one indicates that it is the signature 72 in the entry 70 is a pooled signature obtained by performing a pooling operation on two or more data block signatures 72 the first level of summary 0 which was calculated directly from the data block 8th were calculated. A summary level n indicates that the data block signature was calculated from summarized or data block signatures calculated at the (n-1) th or a previous summary level. For example, a summary level two indicates that the pooled signatures were calculated from the pooled signatures on the first summary level.

According to certain embodiments, a merge operation with signatures 72 in entries 70 executed on consecutive blocks of data 8th in the memory 10 reference 74 so the memory reference 74 for the combined signature on successive data blocks 8th refers.

4th illustrates an example of entries 70 in the deduplication index 16 . The data block entries 80 the first level, the summary level 0, are calculated by calculating the signatures directly from the data blocks 82 generated, which are designated in detail with 82a to 82i. The entries 84 the second level of summarized, the summary level 1 , are calculated by performing a summarization operation (e.g., a multiplication) on the signatures of two entries that refer to successive blocks of data from the previous summarization level 1 refer. The entries 86 the third level of summary, the level of summary 2 , have aggregate signatures that are created by running the Combination operation with the combined signature of two entries 84 of the previous second summary level, which refer to successive data blocks. And finally assign the entries 88 the fourth level of summary to summarize signatures obtained by performing the summarizing operation on the entries 86 of the previous level which refer to successive data blocks. In this way, each further summary level has entries that combine multiple entries that refer to successive records of one or more data blocks to form a combined entry that represents all of the successive data blocks to which the multiple entries refer to Summary operation were subjected.

For example, the deduplication entries refer to 80a and 80b on the data blocks 82a and 82b . The entry 84a is done by summarizing the signatures in the entries 80a and 80b formed, and the entry 84a refers to the successive data blocks 82a and 82b to which the summarized entries 80a and 80b refer to the entry 84a can replace. The deduplication entry 86a is done by summarizing the signatures in the entries 84a and 84b formed so the entry 86a on the successive data blocks 82a , 82b , 82c and 82d refers to which the combined entries 84a and 84b refer to the entry 86a can replace. The deduplication entry 88a is done by summarizing the signatures in the entries 86a and 86b formed so the entry 88a on consecutive data blocks 82a , 82b , 82c , 82d , 82e , 82f , 82g and 82h refers to which the combined entries 86a and 86b refer to the entry 88a can replace. Since the deduplication index 16 to save the deduplication entries 70 If additional storage space is required, storage space can be freed up in this way, so that fewer entries, ie signatures, on the same data blocks 8th in the memory 10 refer.

According to the above embodiments, the merging operation takes place in the merging of two entries on a previous merging level which refer to successive data blocks. According to further embodiments, an operation with more than two entries can take place in a summarization operation, which refer to successive data blocks, so that a combined entry can replace more than two entries and thus refer to the successive data blocks to which more than two combined entries refer. In addition, in certain embodiments, a merge operation can be performed on entries at different levels as long as the entries at different levels refer to consecutive blocks of data.

The aggregation operation with block signatures or aggregate signatures can be a chaining or other operation. In one embodiment, the merge operation can concatenate two m and n byte signatures to form a new merged signature of m plus n bytes. In further embodiments, additional operations can be performed on the concatenated signatures, for example by taking another hash value to generate the final signature. In still other embodiments, the merge operation may include, but are not limited to, other suitable types of operations, including, but not limited to, multiplications, etc.

In embodiments where two entries are merged, the number of data blocks referenced by an entry at a summary level n, (n = 0, 1, 2, 3 ...), can be expressed by the expression 2n.

5 Figure 3 illustrates one embodiment by the memory manager 12 operations performed to insert a data block from a received data object into memory 10 . A data block to be inserted can be a data block of variable length, which can be determined by applying a fingerprint algorithm such as the Rabin fingerprint scheme to the data object to determine the data block boundaries, or it can be a fixed length data block so that the boundaries of the data blocks in the Data object of fixed length can be determined. After receiving the into memory 10 data blocks to be inserted (in block 100 ) is generated by the memory manager 12 (in block 102 ) a signature for the data block and writes it (in block 104 ) into memory 10 . The storage manager 12 adds (in block 104 ) An entry 70 into the deduplication index 16 one that has the signature 72 , a reference to the in memory 10 written data block and a summary level 0 indicates.

6th illustrates one embodiment of operations performed by the memory manager 12 run to entries 70 of the deduplication index 16 summarize. The storage manager 12 starts (in block 120 ) an operation to summarize the entries 70 of the index 16 in response to an event. For example, the memory manager 12 Combine entries if the available space in the memory of the Servers 2 reached a threshold. Alternatively, the memory manager 12 periodically summarize entries after the index 16 has reached a certain size or after a certain period of time. After starting the summary operation (in block 120 ) is determined by the memory manager 14th (in block 122 ) in the index 16 entries to be summarized 70 based on a criterion of the data blocks 8th to which the entries 70 refer so that the entries affected and unaffected by the grouping refer to all data blocks 8th refer in memory to those in data objects 50 are included. The storage manager 12 selects (in block 126 ) several of the entries determined 70 that meet the criterion and on successive data blocks 8th in the memory 10 which should be grouped together. The selected entries to be summarized, which refer to consecutive data blocks, can be on the same or on different summarization levels.

The criterion for selecting the data blocks to be summarized may include an age of the data blocks in the memory, so that data blocks which have been in memory for a long time without being updated are more likely to be selected for summarization than data blocks which have been inserted or updated more recently. Further criteria can be used to select less requested data blocks, whereby the demand for a data block can be determined: the frequency of input / output (I / O) accesses to the data blocks in the memory so that data blocks are more in demand that are accessed more frequently; based on a frequency with which the data blocks appear in the memory in data objects that are inserted into the memory, so that data blocks that appear in multiple data objects are in greater demand; based on a memory area in which the data blocks are stored, so that data blocks that are stored in a more up-to-date area that is accessed more frequently are in greater demand; and on the basis of a data type in the data blocks, data blocks whose data type is more important and which are accessed more frequently can be viewed as being in greater demand.

The storage manager 12 leads (in block 128 ) a summarize operation on the data block signatures 72 in the selected entries 70 to generate a combined signature. The aggregation operation may include multiplying the signature values or performing other operations to determine a new aggregated signature value. In the index 16 becomes (in block 130 ) An entry 70 inserted that contains the combined signature 72 , a reference 74 on the data blocks in memory 10 from which the processed summary and / or data block signature was generated, and a summary level 76 which has a next level from the summary level of the selected entries affected by the summary operation. The selected entries that are affected by the merging are (in block 132 ) removed to be in the deduplication index 16 To create space. If it (in block 134 ) there are multiple entries that have not yet been merged, the process goes back to block 126 to select further identified entries to be combined, otherwise the process is ended.

7a and 7b illustrate an embodiment by the memory manager 12 operations performed to process a received data object using the deduplication index 16 to determine non-duplicate data blocks in the received data object that are in the memory 10 should be inserted. After receiving an object (in block 150 ), the memory manager processes 12 (in block 152 ) the received data object in order to determine data blocks (with a fixed or variable size) of the data in the received data object. To determine blocks of data with variable sizes, the memory manager 12 apply a fingerprint algorithm such as the Rabin fingerprint scheme. Then the memory manager can 12 start a process for calculating signature values for individual data blocks and combinations of data blocks in order to use these with the combined signatures 72 for summarized entries 70 to compare. A summary level counter is (in block 154 ) set to zero. The storage manager 12 determined (in block 156 ) Data block signatures for the determined data blocks by processing the data in the determined data blocks of the received data object, each determined data block signature being associated with the data block that was processed to determine the signature. 8th shows how the memory manager 12 Signature data 190 the data blocks of the received object for each calculated signature and data about the one or more data blocks 194 can manage that of the calculated signature 194 are associated.

After determining the data block signatures for each individual data block in the received object, the memory manager determines 12 (in block 158 ) from the deduplication index 16 a highest level of summary 76 in the entries 70 of the deduplication index 16 , ie the highest level of summary for which signatures have been calculated. If (in block 160 ) the highest determined summary level the value of the counter for Exceeds summary levels, the storage manager runs 12 (in block 162 ) the summarize operation with the summarized or data block signatures 190 which were calculated for the last considered summary level in order to generate a further level of summarized signatures. The storage manager 12 can do the same aggregation operation as creating the deduplication index in 6th to generate a value for a combined signature from at least two other signature values, for example by multiplying the signature values. The counter value for summary levels is (in block 164 ) increases and the process goes back to block 160 to determine whether a next level of summarized signatures should be calculated from the previously calculated summarized signatures at the previous level, which is indicated by the variable of the counter for summarized levels. This is how the storage manager calculates 16 Signature values from received data objects to generate data block signatures from the data blocks in the data object and summarized signatures calculated by performing the summarizing operation (multiplying) with previously calculated summarized or data block signatures.

If the (in block 160 ) determined highest summary level does not exceed the summary counter value, which is after generating summarized and data block signatures for the received data object up to the highest summary level in the deduplication index 16 happens, the process goes (from block 166 ) continue to block 170 in 7b . The storage manager 12 determined (in block 170 ) whether the data block signatures and combined signatures with signatures determined from the data blocks in the received data object 72 in the entries 70 of the deduplication index 16 to match. A match indicates that the data blocks in the received data object that is associated with the summarized or data block signature are copies of successive data blocks 8th that are already in memory 10 are stored, and a mismatched signature indicates that the data blocks associated with the mismatched pooled or data block signature are not copies of data held in memory 10 are stored. If it (in block 172 ) there are mismatched signatures calculated from the summarized or data block signatures of the received data object that do not match signatures in the index 16 match, the data blocks in the received data object, which are associated with the identified non-matching signatures, according to the operations of 5 (in block 174 ) is written to memory. Data blocks in the received data object that are associated with the summarized and data block signatures that match the signatures in the index are not put into memory 10 written as they have copies already in memory 10 represent stored data.

According to the described embodiments, entries of a deduplication index are combined in order to create free space in the index and to enable references to data blocks in the memory in connection with other data blocks to be retained by a combined signature in the deduplication index. If the space in the deduplication index has to be limited, the described embodiments allow data blocks to remain represented in the deduplication index if data is to be avoided from being written twice into the memory. According to certain embodiments, entries of the deduplication index that refer to less frequently accessed data blocks can be combined to make room in the deduplication index to refer to data blocks newly added to memory and at the same time ensure that index entries continue to be deduplicated can refer to these less frequently accessed successive data blocks.

The described embodiments for managing the deduplication index 16 and to summarize entries can be in the server 2 when these data objects are executed by the clients 4th receives. Alternatively, the clients 4th the deduplication index 16 of data in the memory 10 manage to thereby reduce the number of in memory 10 to minimize the data blocks to be saved that go to the server 4th be transmitted. The client 4th can use the deduplication index 16 can be used by performing the operations of 7a and 7b Data blocks in a data object identify the copies of data blocks in memory 10 and then replace the data with the signatures in the index entries for the data blocks referenced in the entries of the deduplication index 16 is referred to over the network 6th to the server 2 to minimize the volume of data transferred.

9 Figure 3 illustrates one embodiment of a computing node 302 , which is an embodiment of the server system 2 and the client systems 4th may have. The server node 2 and the client nodes 4th can be implemented in a cloud computing environment in which the server 2 Storage and deduplication services for the clients 4th provides. The computer node 302 represents only one example of a suitable data processing node and is not intended as a limitation of the To see the scope of use or functionality of embodiments of the invention described herein. Rather, the computer node can 302 be implemented to perform any functionality set forth above.

The computer node 302 can be operated with numerous other environments or configurations of universal or special data processing systems. As examples of well known data processing systems, environments, and / or configurations that are suitable for use with the computer node 302 personal computer systems, server computer systems, thin clients, thick clients, portable or laptop units, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable home electronics, network PCs, small computer systems, large computer systems, and distributed cloud computing environments including any of the above systems or units, and the like.

The computer node 302 may be described generally in connection with executable instructions of a computer system, for example program modules, that are executed by a computer system. In general, program modules can contain routines, programs, objects, components, logic, data structures and so on that carry out certain tasks or implement certain abstract data types. The computer node 302 can be operated in distributed cloud computing environments in which tasks are carried out by remote processing units that are linked to one another via a data transmission network. In a distributed cloud computing environment, program modules can be arranged in storage media from both local and remote computer systems, including in storage units.

The computer node 302 according to 9 is shown in the form of a universal data processing unit. The components of the computer system / server 302 can, without being limited thereto, one or more processors or central processing units 304 , a system memory 306 and a bus 308 that connects different system components, including the system memory 306 with the processor 304 .

The bus 308 Figure 3 illustrates one or more of various types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. Examples are, but not limited to, the ISA bus (industry standard architecture), the MCA bus (micro-channel architecture), the EISA bus (further development of the industry standard architecture), the local VESA bus (video electronics standards Association) and the PCI bus (connection bus for expansion components).

Usually the computer node contains 302 a variety of computer readable media. Such media can be any available media that the compute node is targeting 302 and which contain both volatile and permanent media, exchangeable and permanently installed media and for storing the object data 14th , the deduplication index 16 , the data block data 18th and the program of the memory manager 12 can be used.

The system memory 306 can be computer-readable media in the form of volatile memory such as random access memory (RAM) 310 and / or a cache memory 312 contain. The computer node 302 can also contain further exchangeable / permanently installed, volatile / non-volatile storage media of the computer system. The storage system 313 read from and write to a permanently installed permanent magnetic medium (not shown and commonly referred to as a "hard disk drive"). There may also be a magnetic disk drive, not shown, for reading from and writing to a removable permanent magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from a removable permanent optical disk such as a CD-ROM, DVD-ROM or others optical media and for writing on them. In such cases, each unit can interface with the bus through one or more media 308 be connected. As shown and described in more detail below, the memory 306 contain at least one program product having a set (eg, at least one) of program modules configured to perform the functions of embodiments of the invention.

For example, but not limited to a program / utility 314 , which is a set (at least one) of program modules 316 has, as well as an operating system, one or more application programs, further program modules and program data in the memory 306 be saved. The operating system, the one or more application programs, further program modules and program data or certain combinations thereof can each contain an embodiment of a networked environment. The program modules 316 generally perform the functions and / or practices of embodiments of the invention described herein, such as memory manager operations 12 .

The computer node 302 can also exchange data with one or more external units 318 such as a keyboard, with a pointing device, a screen 320 etc.; with one or more units that allow a user to act on the computer node 302 enable; and / or with any units (e.g. a network card, a modem, etc.) that the computer system / server 302 enable data to be exchanged with one or more other data processing units. Such data exchange can take place via input / output (I / O) interfaces 322 respectively. Furthermore, the computer node 302 via an AC adapter 324 Exchange data with one or more networks such as a local area network (LAN), a general wide area network (WAN) and / or a public network (e.g. the Internet). The power adapter shown 324 exchanges data via the bus 308 with the other components of the computer system / server 302 out. It should be clear that in connection with the computer system / server 302 it is also possible to use other hardware and / or software components that are not shown here. Examples are, but are not limited to: microcode, device drivers, redundant processor units, external disk drive assemblies, RAID systems, tape drives and data archiving storage systems, etc.

Claims (7)

Computer program product for performing a deduplication for a memory, the computer program product having a computer-readable storage medium with a computer-readable program code embodied therein, with the steps: Managing an index of data block signatures calculated from data blocks of data in data objects in the memory, each index entry from the plurality of index entries containing at least one of the data block signatures and a reference to the data block from which the signature was calculated; - selecting entries in the index to be combined by determining entries from the plurality of entries in the index on the basis of a criterion of the data blocks of data referenced in the entries, the criterion ◯ an age of the data blocks in the memory, the age being an amount of time since the data block was inserted into the memory or after it was updated; ◯ and / or a frequency of input / output (I / O) accesses to the data blocks in the memory; ◯ and / or a frequency with which the data blocks appear in the memory in data objects that have been inserted into the memory; ◯ and / or an area of the memory in which the data blocks are stored; ◯ and / or has a data type in the data blocks; Performing a summarization operation on the data block signatures in the selected entries to generate a summarized signature; Inserting an entry in the index containing the summarized signature and a reference to the data blocks in the memory which are referenced in the summarized selected entries; Removing the selected entries from the index which were used to generate the combined signature, the resulting combined entries and entries not affected by the summarization operation referring to all data blocks in the memory which are contained in the data objects managed in the memory; and - Use of the signature index in deduplication operations when inserting data objects into memory. Computer program product according to Claim 1 wherein the selected entries contain references to successive blocks of data in the memory. Computer program product according to Claim 1 wherein the signatures in each of the selected entries comprises a signature calculated from the data block in the memory and / or a combined signature generated by processing previously selected signatures. Computer program product according to Claim 1 wherein the summarizing operation comprises multiplying the signatures in the selected entries. Computer program product according to Claim 1 , wherein the combined signatures generated from the data block signatures define a first summary level in the index, the combined signatures determined from the data block signatures of the received data object having a first summary level of combined signatures of the received data object and the operations further include: Summary level of summarized signatures in the index entries, with each further summarization level of summarized signatures by processing the summarized Forming signatures in entries indicating a previous level of summary; and generating a further summary level of summarized signatures of the received data object for each further summarization level by executing the summarization operation with the summarized signatures of the received data object in at least one previous summarization level, wherein the determination of matching summarized signatures takes into account the summarized signatures that were created by the received data object for all summary levels in the index have been created. A system for performing deduplication on storage, the system comprising: - a processor; and a computer readable storage medium with a programmable code which is executed by the processor to carry out operations, comprising the steps: - Managing an index of data block signatures which have been calculated from data blocks of data in the data objects in the memory, each index entry from the plurality of index entries containing a data block signature and / or a reference to the data block of data from which the signature was calculated; - Selecting entries to be merged in the index by determining entries from the plurality of entries in the index on the basis of a criterion of the data blocks of data referenced in the entries, the criterion being an age of the data blocks in the memory, the Age, a length of time since the data block was inserted into memory or after it was updated; and / or a frequency of input / output (I / O) accesses to the data blocks in the memory; and / or a frequency with which the data blocks appear in the memory in data objects that have been inserted into the memory; and / or an area of the memory in which the data blocks are stored; and / or has a data type in the data blocks; Performing a summarization operation on the data block signatures in the selected entries to generate a summarized signature; Inserting an entry in the index containing the summarized signature and a reference to the data blocks in the memory that are referenced in the summarized selected entries; Removing the selected entries from the index which were used to generate the combined signature, the resulting combined entries and entries not affected by the summarization operation referring to all data blocks in the memory which are contained in the data objects managed in the memory; and - Use of the signature index in deduplication operations when inserting data objects into memory. Procedure for performing deduplication on a store, comprising the steps: Managing an index of data block signatures calculated from data blocks of data in the data objects in the memory, each index entry from the plurality of index entries containing at least one of the data block signatures and a reference to the data block of data from which the signature was calculated; Selecting entries in the index to be summarized by determining entries from the plurality of entries in the index based on a criterion of the data blocks of data referenced in the entries, the criterion being an age of the data blocks in the memory, wherein the age, a length of time since the data block was inserted into memory or after it was updated; and / or a frequency of input / output (I / O) accesses to the data blocks in the memory; and / or a frequency with which the data blocks appear in the memory in data objects that have been inserted into the memory; and / or an area of the memory in which the data blocks are stored; and / or has a data type in the data blocks; Performing a summarization operation on the data block signatures in the selected entries to generate a summarized signature; Inserting an entry in the index containing the summarized signature and a reference to the data blocks in the memory that are referenced in the summarized selected entries; Removing the selected entries from the index which were used to generate the combined signature, the resulting combined entries and entries not affected by the summarization operation referring to all data blocks in the memory which are contained in the data objects managed in the memory; and - Use of the signature index in deduplication operations when inserting data objects into memory.

Images