US8074043B1 - Method and apparatus to recover from interrupted data streams in a deduplication system - Google Patents
Method and apparatus to recover from interrupted data streams in a deduplication system Download PDFInfo
- Publication number
- US8074043B1 US8074043B1 US12/363,207 US36320709A US8074043B1 US 8074043 B1 US8074043 B1 US 8074043B1 US 36320709 A US36320709 A US 36320709A US 8074043 B1 US8074043 B1 US 8074043B1
- Authority
- US
- United States
- Prior art keywords
- data stream
- segment
- unique identifier
- size
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
Definitions
- the present invention relates to the field of data storage, and particularly to providing for data stream interruption while providing data to a single instance storage data store.
- a data deduplication system provides a mechanism for storing a piece of information only one time.
- a piece of information is stored in multiple locations within an enterprise, that piece of information will only be stored one time in a deduplicated backup storage area.
- that piece of information does not change between a first backup and a second backup, then that piece of information will not be stored during the second backup as long as that piece of information continues to be stored in the deduplicated backup storage area.
- Data deduplication can also be employed outside of the backup context thereby reducing the amount of active storage occupied by duplicated files.
- data is divided in a manner that provides a reasonable likelihood of finding duplicated instances of the data. For example, data can be examined on a file-by-file basis, and thus duplicated files (e.g., operating system files and application files and the like) would be analyzed and if the entire file had a duplicate version previously stored, then deduplication would occur.
- duplicated files e.g., operating system files and application files and the like
- a drawback of a file-by-file deduplication is that if a small section of a file is modified, then a new version of the entire file would be stored, including a potentially large amount of data that remains the same between file versions.
- a more efficient method of dividing and analyzing data is to divide file data into consistently-sized segments and to analyze those segments for duplication in the deduplicated data store. Thus, if only a portion of a large file is modified, then only the segment of data corresponding to that portion of the file need be stored in the deduplicated data storage and the remainder of the segments will not be duplicated.
- One mechanism for breaking data into a series of segments is for a client of the deduplication system to provide a stream of data to a deduplication server.
- a stream of data can include numerous data objects (e.g., backed-up files).
- the deduplication system can select an appropriate segment size and store data from the incoming data stream into a series of appropriately sized segments.
- a potential problem with such a scheme of breaking a data stream into segments is that a data stream may abnormally terminate during the course of providing data to a segment. Such an abnormal termination may result in the last segment of that transmission being incomplete.
- a data deduplication system it is therefore desirable for a data deduplication system to have a stream segmenter that can associate a data stream received after an abnormal termination of a previous data stream with that previous data stream in order to determine an appropriate segment size for the remainder of a data object received at the beginning of the second data stream. Further, it is desirable for the stream segmenter of the deduplication server to perform a segment splice, allowing fixed size segmentation of the data object to proceed at the proper segment alignment for deduplication to occur, as if the first data stream had never been interrupted.
- Embodiments of the present invention provide a method, apparatus and computer-readable storage medium having instructions for detection and proper deduplication of a re-started data stream in a segmentation analysis-based deduplication system by retaining information about a previous data stream and using that information when performing segmentation of the re-started data stream.
- One embodiment of the present invention provides such data stream re-start capability by determining whether a received data stream associated with a unique identifier is the first data stream associated with the unique identifier.
- a segment of memory to contain a first amount of data from the data stream is allocated, the size of which is determined either from an identification of a type of data object received in the current data stream or a size of a last segment of a previous data stream associated with the unique identifier, if the current data stream is not the first data stream associated with the unique identifier.
- One aspect of the above embodiment determines whether the current data stream is the first data stream associated with a unique identifier by receiving a sequence number associated with the current data stream and the unique identifier and comparing the unique identifier/sequence number pair of the current data stream against unique identifier/sequence number pairs of previously received and terminated data streams.
- a further aspect of this embodiment can store the unique identifier/sequence number pair of the current data stream for future comparison against subsequently received data stream unique identifier/sequence number pairs.
- Another aspect of this embodiment can store the size of the segment with the unique identifier/sequence number pair for the data stream.
- Another aspect of the above embodiment of the present invention determines the size of the segment by reading the first data object type from the current data stream and selects a segment size corresponding to the first data object type, if the current data stream is the first data stream associated with the unique identifier. If the current data stream is not the first data stream associated with the unique identifier, then this aspect of the above embodiment of the present invention determines the segment size of a last data object of the previous data stream associated with the unique identifier. A further aspect of the above embodiment determines the segment size for the first segment of the re-started data stream by subtracting the size of the last segment of the previous data stream associated with the unique identifier from the segment size of the last data object of the previous data stream associated with the unique identifier.
- FIG. 1 is a simplified block diagram illustrating a method of data deduplication usable in conjunction with embodiments of the present invention.
- FIG. 2 is a simplified block diagram of a network incorporating a deduplication system usable in conjunction with embodiments of the present invention.
- FIG. 3 is a simplified block diagram illustrating components of deduplication server 240 .
- FIG. 4A is a simplified block diagram illustrating an example of information stored by metadata database engines and content routers of a deduplication system usable in conjunction with embodiments of the present invention.
- FIG. 4B is a simplified block diagram illustrating information stored in the event of a change of one of the files illustrated in FIG. 4A .
- FIG. 5 illustrates varying segment sizes for different data objects in a data stream as employed with embodiments of the present invention.
- FIG. 6 is a simplified illustration of data shifting in segments that can occur in the event of an abnormal termination of a data stream.
- FIG. 7 is a simplified flow diagram illustrating steps performed in a “splicing” operation, in accord with one embodiment of the present invention.
- FIG. 8A is a simplified flow diagram illustrating steps involved with merging the last segment from a first data stream with the first segment from a second data stream, in accord with embodiments of the present invention.
- FIG. 8B is a simplified block diagram illustrating merging of segments as provided by the process of FIG. 8A .
- FIG. 9 is a simplified block diagram illustrating one embodiment of a stream segmenter 350 usable in embodiments of the present invention.
- FIG. 10 depicts a block diagram of a computer system suitable for implementing aspects of the present invention.
- FIG. 11 is a block diagram depicting a network architecture suitable for implementing aspects of the present invention.
- Embodiments of the present invention provide for detection and proper deduplication of a re-started data stream in a segmentation analysis-based deduplication system by retaining information about a previous data stream and using that information when performing segmentation of the re-started data stream.
- Embodiments of the present invention can retain information such as a unique identifier of the previous data stream, an indicator of whether the previous data stream was the first or one of a sequence of data streams associated with the unique identifier, a segment size associated with the last data object received in the previous data stream, and a record of how much data was present in the last segment associated with the previous data stream.
- Embodiments of the present invention can then use the information from the previous data stream in determining whether a second data stream (e.g., the re-started data stream) is associated with the previous data stream (e.g., has the same unique identifier) and is the next data stream in the sequence of data streams associated with that unique identifier. If the second data stream is associated with the previous data stream, then embodiments of the present invention can use the retained segment size information to set a segment size for the first data object of the second data stream, and use the size of last segment information to determine how much information should be put in the first segment associated with the second data stream in order to maintain proper alignment of the remainder of the segments for the first data object in the second data stream to allow for proper deduplication.
- a second data stream e.g., the re-started data stream
- the previous data stream e.g., has the same unique identifier
- embodiments of the present invention can use the retained segment size information to set a segment size for the first data object of the second data stream
- a general concept behind data deduplication is to reduce an amount of storage needed for enterprise data by eliminating duplicated data content.
- One embodiment of a deduplicated data storage area is a single instance storage.
- a single instance storage volume only a single instance of a piece of data is stored.
- a common use of a single instance storage is in maintaining data backups for servers and other computing clients in a network. For each backup, only a single instance of information duplicated in the network will be stored in the single instance storage area.
- data items that have not changed from one backup to another need not be stored in the subsequent backup. In this manner, significant savings in data storage space can be realized.
- an initial full backup of a file server can be performed in which the data included in the backup is divided and identified in a manner as discussed below.
- Subsequent daily backups of the file system involve identifying new or changed segments of data using the same data identification system as performed with the full backup and then storing only the new or changed data in the single instance storage area.
- a reduction in the size of the backup subsequent to the initial full backup can be as much as 99%.
- the single instance storage approach to backups allows for very small daily backups with an ability to recover a full image from any backup on any day.
- the traditional distinction between a “full” backup and an “incremental” backup disappears since, although only new or changed data is saved in the single instance storage area, all the data that is backed up can be restored at any time using a single backup restore.
- FIG. 1 is a simplified block diagram illustrating a method of data deduplication.
- Files 110 and 120 share identical content, but the files themselves are not completely identical.
- Each file is broken into segments: A, B, C, D, E for file 110 and A, B, Q, D, L for file 120 .
- the data deduplication system will write file segments A, B, C, D, E, Q and L into the single instance storage area ( 130 ) and will not write file segments A, B and D from file 120 into the single instance storage area.
- two pointers e.g., one corresponding to file 110 and the other to file 120
- metadata containing, for example, pointers to the data of a file can be stored in a metadata storage area.
- FIG. 2 is a simplified block diagram of a network incorporating a deduplication system usable with embodiments of the present invention.
- Client computers 210 are coupled to a network 220 .
- Each client computer has access to one or more file systems that are stored in either storage volumes directly-coupled to each of the clients or by storage volumes that are served to the client computers by a storage area network (SAN), network attached storage (NAS), storage appliance and the like (not shown).
- Client computers can include, for example, servers and desktop nodes.
- Embodiments of network 220 can include, for example, a local area network, metro area network, wide area network, or any combination thereof.
- Backup server 230 is also coupled to network 220 .
- Backup server 230 is configured to, for example, manage administrative tasks related to backing up clients 210 . Such tasks include communicating with clients 210 to initiate backup tasks on the clients, maintaining databases related to files and other information backed up from file systems associated with the clients, and managing or tracking resources storing backups of clients 210 .
- backup server 230 is further configured to communicate with deduplication server 240 for purposes of storing backups of client 210 in resources controlled by deduplication server 240 . Such communication can be via network 220 or via a direct link between the backup server and the deduplication server.
- Information that can be provided by backup server 230 to deduplication server 240 can include a unique identification associated with each data stream provided by clients 210 to the deduplication server.
- the backup server can also provide sequence number identification for each uniquely identified data stream. Deduplication server 240 can then use such information to associate received data streams from clients in accord with embodiments of the present invention, as will be discussed more fully below.
- Deduplication server 240 is coupled to network 220 and performs a variety of tasks related to management and implementation of deduplication services for the system illustrated in FIG. 2 .
- Deduplication server 240 can include one or more physical servers configured to perform a variety of tasks related to deduplication.
- Deduplication server 240 is in turn coupled to a storage pool for deduplicated data that includes one or more metadata volumes 250 and one or more content volumes 260 .
- FIG. 3 is a simplified block diagram illustrating components of deduplication server 240 .
- Storage pool authority module 310 performs administration tasks related to management of the storage pool resources. Among such administrative tasks include setting policies for storage resources, such as retention times for data stored, types of metadata associated with stored data that is required, size of data segments, and the like. Storage pool authority module 310 can also perform scheduling-related tasks, management of concurrently arriving data streams, and management of data being provided to content storage (e.g., locations for storing various incoming data). Storage pool authority module 310 can store configuration information for the storage pool in a configuration database 315 .
- Deduplication server 240 also includes a metadata database server module 320 that communicates with storage pool authority 310 or with data sources (e.g., clients 210 or backup server 230 ).
- Metadata database server module 320 is configured to distribute metadata received from storage pool authority 310 or the data sources to one or more metadata database engines 325 ( 1 )-(M). Such metadata includes information about the nature of the data stored by the storage pool. The metadata stored by metadata database engines 325 ( 1 )-(M) will be discussed in greater detail below.
- Deduplication server 240 also includes one or more content routers 330 ( 1 )-(N) that are communicatively coupled to storage pool authority 310 and to data sources (e.g., clients 210 or backup server 230 ).
- the content routers are configured to receive data content being stored in the storage pool managed by deduplication server 240 and to manage the storage of that data in one or more content storage volumes 340 .
- Storage pool authority module 310 can be configured to distribute data in a fair and efficient manner across all content routers 330 ( 1 )-(N).
- Deduplication server 240 also includes stream segmenter module 350 that can receive data streams and other communication from the various data sources (e.g., clients 210 and backup server 230 ) and perform tasks related to segmenting in the incoming data streams in preparation for deduplication storage, as will be discussed more fully below.
- Stream segmenter 350 can also receive the identification information and other information related to previous data streams provided by the backup server or the storage pool authority 310 , as will be discussed more fully below.
- modules 310 , 320 , 330 and 350 can be executed by one or more physical servers configured to perform the functionality required by the various modules.
- FIG. 4A is a simplified block diagram illustrating an example of information stored by the metadata database engines and the content routers of a deduplication system capable of implementing embodiments of the present invention.
- FIG. 4A illustrates client files 410 , 420 and 430 , each of which contains identical content.
- Files 410 , 420 and 430 can be located on the same client or on different clients.
- Each file is associated with metadata that describes the file ( 412 , 422 , 432 ), including, for example, the filename, size of the file, and a modification date.
- Each file also contains data content associated with the file ( 415 , 425 , 435 ).
- Identity of the content of the files is established by a fingerprint generated using the content of each file ( 417 , 427 , 437 ). The fingerprint of the file content is generated, for example, using a hash or checksum process sufficient to uniquely identify the file content.
- a deduplication process can proceed in the following manner. If, for example, file 410 , 420 and 430 are being backed up, file 410 is provided to deduplication server 140 . Metadata 412 associated with file 410 is provided to a metadata database engine 325 along with the associated fingerprint 417 . Content 415 of file 410 is divided into a set of segments A, B, C, D and E ( 450 ) (e.g., by stream segmenter 350 ) and these segments are provided to a content router 330 for storage in a content storage volume 340 along with the associated fingerprint 417 as a storage object 440 . Thus, the unique fingerprint 417 serves as a link between the information stored in the metadata database and the file content file stored in the content storage volume. In one embodiment of the present invention, fingerprint 417 is generated by first determining a separate segment fingerprint for each of segments A, B, C, D and E using a hashing computation and then combining or hashing those segment fingerprints values to result in fingerprint 417 .
- Metadata 422 is provided to the metadata database 325 along with fingerprint 427 .
- Deduplication server 140 checks to see if data associated with fingerprint 427 has already been stored in content storage volume 340 . Since storage object 440 has already been provided to content storage volume 340 , that data is not duplicated in the content storage volume. A similar process occurs for file 430 in which metadata 432 and the associated fingerprint 437 are stored in the metadata database but no duplication of data is made in content storage volume 340 .
- FIG. 4B is a simplified block diagram illustrating information stored in the event of a change of one of the files illustrated in FIG. 4A .
- FIG. 4B illustrates a change occurring in file 420 creating a file 420 ′.
- Metadata 462 associated with file 420 ′ reflects a new modification date.
- Content 465 is illustrated showing that original segments C and E of file 420 have been changed to segments O and P, while the other segments remain the same. Since content 465 is different from original content 425 , a different fingerprint 467 is generated.
- file metadata 462 is provided to metadata database 325 along with new fingerprint 467 .
- file content 465 is provided to a content router associated with content storage volume 340 .
- the content router can establish that of all the segments comprising file content 465 , only segments O and P have yet to be stored in the content storage volume and fingerprint 467 is associated with the shared previously stored segments and the modified segment.
- One example of how a content router can determine whether a data segment has been previously stored is by maintaining checksums of each stored data segment and comparing a checksum of a data segment to be stored against those already stored. That association of fingerprint 467 and the previously stored and modified segments is stored as a storage object 470 .
- the above discussion illustrates a process of dividing a file into segments and providing those segments to a single instance storage system for deduplication.
- Deduplication of data in a content storage volume by a content router is then carried out on a segment-by-segment basis. This allows for changes in part of a file to be reflected by storing only those segments that have been modified.
- a default segment size may be chosen for a volume in a manner to optimize management of information and to increase the likelihood of duplicate segments. But such a default segment size may not provide for optimal management of data in the deduplication server. For example, if the default segment size is too small, then a deduplication server can be overwhelmed by having to manage metadata and data for a large number of segments.
- embodiments of the present invention provide for varying segment sizes based upon a type of data object being stored in the deduplication server. This allows for choosing an optimal segment size for a particular type of file. For example, a file object containing sound or video data will have a low likelihood of having duplicate data segments with a second file unless the second file is a copy of the first file. In such a case, a large segment size (e.g., one or more megabytes) may be reasonable.
- other types of data objects may include data that is largely static but also has parts of which may be altered (e.g., text documents and database storage files).
- a smaller segment size e.g., 128 k bytes
- segment size is reasonable to capture both the static sections of data and the changing sections of data in such a way that there is a high probability of duplicate segments with those previously stored in the deduplication server.
- FIG. 5 illustrates the concept of varying segment sizes for different data objects in a data stream.
- the stream segmenter 350 selects a segment size based upon the type of object of File 1 .
- stream segmenter 350 chooses a default segment size for File 1 of 128 k bytes.
- Segments 1 - 4 of File 1 are each of the default segment size and are filled with data from the data stream.
- Segment 5 is filled with data until the beginning of data object File 2 is received in the data stream. At that point, segment 5 is closed containing the amount of data received to complete File 1 (e.g., as illustrated, 60 k bytes).
- the stream segmenter determines the type of data object of File 2 , and selects a segment size of 512 k bytes for the segments containing data from File 2 . Segments 6 and 7 , as illustrated, are each of 512 k bytes. Once again, when the data object containing File 3 is received, the stream segmenter interprets the type of data object and selects a segment size of 2 megabytes. The process of receiving data objects in a data stream and placing that data into appropriately sized segments continues until the data stream transmission is terminated. Upon termination of the data stream transmission, the last segment being filled is closed.
- the segments are provided to content routers 330 for analysis and storage within the deduplication server's content stores 340 .
- the stream segmenter can be programmed with expected data object types and corresponding segment sizes for those expected data object types. Further, the stream segmenter can have a default segment size in the event that a received data object does not have a preprogrammed optimal segment size.
- a data stream from a client may be terminated prematurely or abnormally due to, for example, a network connection failure, a client crash, and the like.
- stream segmenter 350 will close the final segment of the terminated data stream as it would for a normal termination, as discussed above.
- the stream segmenter may receive data from the middle of a data object and from the middle of a segment. Therefore, stream segmenter 350 will be unable to select an optimal segment size for the data because it will be unable to determine the type of object being received.
- data would be inserted at the beginning of a new segment and filling that segment and then moving on to the next segment. Thus, subsequent data in the data segments would be shifted from originally intended locations by an amount equivalent to the data found in the last segment of the abnormally terminated data stream.
- FIG. 6 is a simplified illustration of the shifting that can occur in the event of an abnormal termination of a data stream.
- a data stream data object 610 is received and without an abnormal termination is broken into a segment stream 620 .
- the data stream is broken into two data streams, DS 1 and DS 2 , due to an abnormal termination.
- DS 1 is broken up into segments, and the data in those segments aligns with that in segment stream 620 (e.g., data in segments 1 - 3 of segment stream 640 is the same as corresponding segments in segment stream 620 ).
- segment 4 of segment stream 640 is not the same as segment 4 of segments 620 filled from the unterminated data stream.
- a new segment is formed to include the data received in DS 2 .
- the new segment does not contain the same data as segment 5 from segments 620 because it has been shifted by the amount received in DS 1 for segment 4 . Similarly, all subsequent segments will be shifted by that amount. Since the data in the segments is shifted, the segments do not contain the same data as those previously received and stored segments in the deduplication server and therefore those segments are not eligible for deduplication even though they were formed from the same data stream data object previously received in segment stream 620 (e.g., during a previous backup).
- stream segmenter 350 can perform a “splicing” operation such as that illustrated by segment stream 650 .
- segments 1 - 3 are the same as those in segment stream 620 .
- segment 4 a is prematurely closed and therefore does not match segment 4 of segment stream 620 .
- stream segmenter 350 can form a reduced size data segment segment 4 b that contains the data that would have completed segment 4 of data stream 620 .
- Segments 5 and beyond are formed of the selected size for the data object and contain the same data as segments 5 and beyond of segment stream 620 .
- Such splicing makes at least segments 5 and above eligible for deduplication processing.
- stream segmenter 350 associates a re-started data stream (DS 2 ) with a previous data stream that abnormally terminated (DS 1 ).
- a backup server or other central authority, can provide the clients with a unique identifier for each data stream transmitted from a client to the deduplication server.
- a unique identifier can be generated in a number of ways, for example, using a date and time of day for a random number generator seed.
- the unique identifier of a data stream can be provided to a deduplication server upon initiation of the data stream to the deduplication server from the client.
- the backup server or other central authority can track a sequence number for a data stream associated with a unique identifier. That is, if a data stream is the first data stream associated with a unique identifier, then a sequence number associated with that data stream is one. If a data stream associated with a unique identifier is abnormally terminated, a re-started data stream associated with that unique identifier will be given a sequence number of two, and so on until the data stream associated with the unique identifier normally terminates.
- the sequence number associated with a data stream can also be provided to the deduplication server by the client at the time of initialization of transmission of the data stream to the deduplication server.
- the unique identifier and sequence number of the data stream can then be examined to determine whether the current data stream is a resumption of a previously terminated data stream.
- stream segmenter 350 or another part of the deduplication server e.g., storage pool authority 310
- the unique identifier and sequence number of that data stream can be compared with the saved unique identifier and sequence number pairs to determine whether the new data stream is associated with a previously terminated data stream. If so, then the stream segmenter can perform subsequent operations taking that association into account.
- a segment size for a data object can be determined by the stream segmenter at the beginning of receipt of a data object. Segment size is dependent upon the type of data object. If a data stream begins mid-data object, the stream segmenter cannot determine an appropriate segment size. Thus, another piece of information that the stream segmenter can retain from a terminated data stream is the selected segment size for the last data object of that data stream. Then, when the data stream re-starts mid-data object, that stored segment size will be used for remaining data associated with that data object in the re-started data stream.
- the stream segmenter further retains the amount of data placed into the last segment of the previous data stream. Using this information, stream segmenter 350 can then determine an amount of data necessary to complete a data segment of the selected size (e.g., by subtracting the amount of data actually put into the last segment of the previous data stream from the selected segment size) and generate a first segment for the re-started data stream of that smaller length. Once that initial segment is completed, subsequent segments can then be of the selected length for the data object. In this manner, as illustrated by segment stream 650 in FIG. 6 , segments subsequent to the initial segment will align with segments expected from a uninterrupted data stream (e.g., segment stream 620 ).
- segment stream 650 segments subsequent to the initial segment will align with segments expected from a uninterrupted data stream (e.g., segment stream 620 ).
- FIG. 7 is a simplified flow diagram illustrating steps performed in a “splicing” operation, in accord with one embodiment of the present invention.
- a stream segmenter 350 receives a data stream from a client 210 ( 705 ).
- a unique identifier associated with the data stream is then read ( 710 ).
- the unique identifier can be provided by a central authority that tracks all identifiers for data streams from clients to a deduplication server. Each data stream associated with a flow of data from a client is uniquely identified from all others.
- a sequence number associated with the data stream is read ( 715 ). Again, as discussed above, a sequence number can be assigned or tracked by a central authority, and relates to the sequence of the current data stream associated with the unique identifier.
- the unique identifier and sequence number can be stored by the stream segmenter or in another associated module in the deduplication server ( 720 ). A determination can then be made as to whether the current data stream is first in a sequence associated with the unique identifier ( 725 ). Such a determination can be performed by determining whether the sequence number is greater than a predetermined baseline sequence number. Alternatively, a comparison can be performed between the unique identifier/sequence number pair and previous unique identifiers/sequence numbers stored by the stream segmenter or recalled from the deduplication server (e.g., the storage pool authority).
- the deduplication server e.g., the storage pool authority
- the stream segmenter locates a data object boundary in the data stream ( 730 ). Based upon the type of data object, a segment size is determined ( 735 ) (e.g., by using a lookup table). The determined segment size will then be used for segments associated with the data object. The determined segment size is then stored in association with the unique identifier and sequence number of the data stream ( 740 ). Segments are then generated containing data stream data using the determined segment size starting at the data object boundary ( 745 ). The segments are provided to the deduplication server (e.g., a content router 330 ) for processing and possible storage by the deduplication server. If the data stream is not terminated, the process of placing data stream data in segment sizes is continued until the next object boundary in the data stream and then the process repeats ( 750 ).
- the deduplication server e.g., a content router 330
- the last segment containing data from the data stream is closed ( 755 ).
- the size of the last segment is stored in association with the unique identifier and sequence number of the data stream ( 760 ).
- the last segment is then provided to the deduplication server for processing and storage ( 765 ).
- the stream segmenter can then await a next data stream to process.
- a data stream is not the first sequence number associated with a unique identifier ( 725 )
- the data stream is treated as a re-started data stream.
- the information stored related to the previous sequence number associated with the unique identifier is retrieved in order to determine the selected size of segments for the current data object and the size of the last segment stored in association with the previous sequence number.
- a determination is made as to whether the last segment size is the same as the selected segment size for the data object ( 770 ). If the last segment size is not equal to the selected segment size, then a first segment is generated of a size equal to the selected segment size minus the size of the last segment of the previous sequence number of the data stream ( 775 ).
- That first segment is provided to the deduplication server for processing ( 780 ). Subsequent segments can then be generated to contain data stream data using a segment size equal to the selected segment size until the next data object boundary is reached in the data stream ( 785 ). Subsequent segments are provided to the deduplication server for processing ( 790 ). Once the data object boundary is reached, processing occurs as discussed above in steps 730 - 765 .
- two separate segment streams are tracked by the deduplication server. All the segments of the two segment streams can be analyzed for deduplication by the deduplication server against previously stored segments associated with other stored versions of the data object that was being transmitted at the time of abnormal termination of the first data stream. But the segment that was being formed at the time of abnormal termination of the first data stream and the truncated segment beginning the storage of the second data stream will not be able to be available for deduplication against a previously stored version of that segment from an uninterrupted data stream (e.g., compare segment 4 of segment stream 620 with segments 4 A and 4 B of segment stream 650 ).
- segments 4 A and 4 B of segment stream 650 are combined into a single segment that can then be analyzed by the deduplication server.
- FIG. 8A is a simplified flow diagram illustrating steps involved with merging the last segment from a first data stream with the first segment from a second data stream. The steps illustrated in FIG. 8A can be performed in place of steps 775 and 780 of FIG. 7 .
- stream segmenter 350 can allocate a segment of a size equal to the selected segment size ( 810 ). Data from the last segment associated with the previous sequence number of the data stream is then recalled from the deduplication server ( 820 ) and inserted into the beginning of the current segment ( 830 ). The current segment is then completed with data from the current sequence number of the data stream ( 840 ).
- the current segment can then be provided to the deduplication server for processing ( 850 ).
- processing includes associating the first portion of the current segment with the previous sequence number of the data stream ( 860 ) and associating the second portion of the current segment with the current sequence number plus an offset equaling to the stored last segment size from the previous sequence number of the data stream ( 870 ). Processing then continues as presented in FIG. 7 at element 785 .
- a single segment is created that bridges data from the previous sequence number of the data stream and the current sequence number of the data stream. That is, for example, segments 4 A and 4 B from segment stream 650 are combined into a single segment which can then be compared for deduplication purposes against previously stored segment 4 of segment stream 620 .
- the deduplication server can then track that the first part of the merged segment is associated with the first sequence number of the data stream by modifying pointers associated with the first sequence of the data stream to related segments.
- the subsequent sequence number of the data stream is then tracked by the deduplication server as beginning with the merged segment at an offset point within that segment at which the data from the subsequent sequence number began.
- FIG. 8B is a simplified block diagram illustrating merging of segments as provided in FIG. 8A .
- a data stream 875 is interrupted and resumed, resulting in two data streams DS 1 and DS 2 .
- DS 2 resumes data stream 875 at the point which DS 1 was terminated.
- Segment stream 880 illustrates segments that would have been formed had data stream 875 not abnormally terminated (e.g., as would be stored in the single instance data store from a previous backup data stream).
- Segments Seg 0 and Seg 1 are illustrated as long segments generated in response to, for example, a video or audio data object for which the only duplicate segments would be expected for a copy of the video or audio data object.
- Segment stream 885 illustrates segments formed in response to the interrupted data stream 875 . Seg 2 ends at the point in which data stream DS 1 terminated. Seg 3 begins at the point in which data stream DS 2 begins. As illustrated, neither Seg 2 nor Seg 3 in segment stream 885 can be analyzed for deduplication against Seg 0 stored from segment stream 880 . In addition, data in segments following Seg 3 in segment stream 885 will be shifted and therefore not eligible for deduplication against segments stored from a segment stream 880 .
- Segment stream 890 illustrates a result of employing the steps of FIG. 8A on the abnormally terminated data stream 875 .
- a single, merged segment Seg 2 / 4 is generated that contains the data from Seg 2 , which was stored when DS 1 terminated, and data from the beginning of DS 2 .
- Merged segment Seg 2 / 4 is the same length as Seg 0 from segment stream 880 , and is therefore eligible to be processed for deduplication against a previously stored version of Seg 0 .
- subsequent segment Seg 1 of segment stream 890 can be checked against previously stored versions of Seg 1 of, for example, segment stream 880 because there is no shifting of data in segments following the merged segment.
- stream segmenter tasks can be performed either by a backup agent executing on a client 210 or by deduplication server 240 or a specific module within deduplication 640 or a separate compute node configured to receive data streams from the clients and then provide segment streams to the deduplication server.
- FIGS. 7 and 8A illustrate alternative embodiments for allocating data from a re-started data stream to a first data segment associated with the re-started data stream.
- Embodiments of the present invention are not limited to only one of the embodiments illustrated in FIG. 7 or 8 A but can incorporate both. For example, if a determination is made that a last segment size associated with a previous sequence number of a data stream is not equal to the selected segment size ( 770 ), then a determination can be made as to whether to follow the method illustrated in elements 775 - 780 of FIG. 7 or 810 - 870 of FIG. 8A . Such a determination can be based upon, for example, the selected segment size for data from the current data object.
- the mechanism illustrated by FIG. 8A can be selected. Any segment sizes below that threshold can be subject to the mechanisms illustrated in FIG. 7 . In this manner, the mechanism of FIG. 7 , which allows for a less burdensome tracking of information related to a merged (or “spliced”) data segment is performed where one would not expect significant negative repercussions for having truncated segments stored in the deduplication server. But for large segment sizes, burdening the deduplication server with non-deduplicatable truncated segments can be avoided.
- FIG. 9 is a simplified block diagram illustrating one embodiment of a stream segmenter 350 usable in accord with the present invention.
- Stream segmenter 350 incorporates one or more input network interfaces 910 that are configured to receive data streams from clients 210 . Each network interface can receive one or more data streams at a time, and stream segmenter 350 can process multiple incoming data streams simultaneously.
- Data from the incoming data streams is provided to a stream director module 920 which provides data from each incoming data stream to a corresponding selected segment buffer in segment buffer memory 940 .
- information about the data stream e.g., unique identifier and sequence number
- data objects in the data stream is provided to processor 930 .
- Processor 930 can determine whether a stream is a first in a sequence for a unique identifier. If so, then the processor can examine a type of a data object received in the data stream and select an appropriate size for a segment for that data object type. An appropriate memory size to store a segment of the selected segment size in segment buffer memory 940 . Segment buffer memory can be an amount of memory from which smaller segment buffers are dynamically allocated according to the determined segment sizes for the various data stream data objects received from the clients (e.g., SB 1 , SB 2 , SB 3 , SB 4 , etc.). Stream director 920 then directs the data from each data stream to the appropriate dynamic segment buffer for that data stream.
- segment buffer memories can be associated with each input network interface 910 within which the appropriate segment size for the incoming data objects are allocated.
- An advantage of the first approach is that buffer memory space can be more efficiently allocated because each segment buffer does not need to be configured initially to hold the largest possible segment size, a portion of which would then be wasted if smaller segment sizes are needed for a particular data object.
- a segment buffer (e.g., SB 1 ) is filled with incoming data from a data stream
- the segment data is then passed to an output segment stream selector module 950 , which can format the segment and provide the segment to an output network interface 960 .
- the output segment stream selector can provide header information for each segment, such as a data stream unique identifier or sequence number of the segment.
- Output network interfaces 960 then provide the segment stream to an appropriate content router 330 . Determination of the appropriate content router is provided by deduplication server 240 and communicated to processor 930 .
- Embodiments of the present invention employing the above-described mechanism for processing related to an abnormally terminated data stream address the issues of shifted data segments for a re-started data stream. By doing so, embodiments of the present invention avoid storage of duplicate data within a deduplication server by enhancing a probability that duplicate segments may be found for a data object previously stored within the deduplication server. Furthermore, the mechanism of merging truncated segments from the interrupted data stream and the re-started data stream enables a deduplication server to compare the merged segment with segments from a previously stored data object in the deduplication server.
- the present invention can be implemented using a variety of computer systems and networks.
- An example of one such computing and network environment is described below with reference to FIGS. 10 and 11 .
- FIG. 10 depicts a block diagram of a computer system 1010 suitable for implementing aspects of the present invention (e.g., clients 210 , backup server 230 , and deduplication server 240 ).
- Computer system 1010 includes a bus 1012 which interconnects major subsystems of computer system 1010 , such as a central processor 1014 , a system memory 1017 (typically RAM, but which may also include ROM, flash RAM, or the like), an input/output controller 1018 , an external audio device, such as a speaker system 1020 via an audio output interface 1022 , an external device, such as a display screen 1024 via display adapter 1026 , serial ports 1028 and 1030 , a keyboard 1032 (interfaced with a keyboard controller 1033 ), a storage interface 1034 , a floppy disk drive 1037 operative to receive a floppy disk 1038 , a host bus adapter (HBA) interface card 1035 A operative to connect with a Fibre Channel network 1090 ,
- mouse 1046 or other point-and-click device, coupled to bus 1012 via serial port 1028
- modem 1047 coupled to bus 1012 via serial port 1030
- network interface 1048 coupled directly to bus 1012 .
- Bus 1012 allows data communication between central processor 1014 and system memory 1017 , which may include read-only memory (ROM) or flash memory (neither shown), and random access memory (RAM) (not shown), as previously noted.
- the RAM is generally the main memory into which the operating system and application programs are loaded.
- the ROM or flash memory can contain, among other code, the Basic Input-Output system (BIOS) which controls basic hardware operation such as the interaction with peripheral components.
- BIOS Basic Input-Output system
- Applications resident with computer system 1010 are generally stored on and accessed via a computer-readable medium, such as a hard disk drive (e.g., fixed disk 1044 ), an optical drive (e.g., optical drive 1040 ), a floppy disk unit 1037 , or other storage medium. Additionally, applications can be in the form of electronic signals modulated in accordance with the application and data communication technology when accessed via network modem 1047 or interface 1048 .
- Storage interface 1034 can connect to a standard computer-readable medium for storage and/or retrieval of information, such as a fixed disk drive 1044 .
- Fixed disk drive 1044 may be a part of computer system 1010 or may be separate and accessed through other interface systems.
- Modem 1047 may provide a direct connection to a remote server via a telephone link or to the Internet via an internet service provider (ISP).
- ISP internet service provider
- Network interface 1048 may provide a direct connection to a remote server via a direct network link to the Internet via a POP (point of presence).
- Network interface 1048 may provide such connection using wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like.
- CDPD Cellular Digital Packet Data
- FIG. 10 Many other devices or subsystems (not shown) may be connected in a similar manner (e.g., document scanners, digital cameras and so on). Conversely, all of the devices shown in FIG. 10 need not be present to practice the present invention.
- the devices and subsystems can be interconnected in different ways from that shown in FIG. 10 .
- the operation of a computer system such as that shown in FIG. 10 is readily known in the art and is not discussed in detail in this application.
- Code to implement the present invention can be stored in computer-readable storage media such as one or more of system memory 1017 , fixed disk 1044 , optical disk 1042 , or floppy disk 1038 .
- the operating system provided on computer system 1010 may be MS-DOS®, MS-WINDOWS®, OS/2®, UNIX®, Linux®, or another known operating system.
- a signal can be directly transmitted from a first block to a second block, or a signal can be modified (e.g., amplified, attenuated, delayed, latched, buffered, inverted, filtered, or otherwise modified) between the blocks.
- a signal can be directly transmitted from a first block to a second block, or a signal can be modified (e.g., amplified, attenuated, delayed, latched, buffered, inverted, filtered, or otherwise modified) between the blocks.
- modified signals e.g., amplified, attenuated, delayed, latched, buffered, inverted, filtered, or otherwise modified
- a signal input at a second block can be conceptualized as a second signal derived from a first signal output from a first block due to physical limitations of the circuitry involved (e.g., there will inevitably be some attenuation and delay). Therefore, as used herein, a second signal derived from a first signal includes the first signal or any modifications to the first signal, whether due to circuit limitations or due to passage through other circuit elements which do not change the informational and/or final functional aspect of the first signal.
- FIG. 11 is a block diagram depicting a network architecture 1100 in which client systems 1110 , 1120 and 1130 , as well as storage servers 1140 A and 1140 B (any of which can be implemented using computer system 1010 ), are coupled to a network 1150 .
- Storage server 1140 A is further depicted as having storage devices 1160 A( 1 )-(N) directly attached, and storage server 1140 B is depicted with storage devices 1160 B( 1 )-(N) directly attached.
- Storage servers 1140 A and 1140 B are also connected to a SAN fabric 1170 , although connection to a storage area network is not required for operation of the invention.
- SAN fabric 1170 supports access to storage devices 1180 ( 1 )-(N) by storage servers 1140 A and 1140 B, and so by client systems 1110 , 1120 and 1130 via network 1150 .
- Intelligent storage array 1190 is also shown as an example of a specific storage device accessible via SAN fabric 1170 .
- modem 1047 , network interface 1048 or some other method can be used to provide connectivity from each of client computer systems 1110 , 1120 and 1130 to network 1150 .
- Client systems 1110 , 1120 and 1130 are able to access information on storage server 1140 A or 1140 B using, for example, a web browser or other client software (not shown).
- client software not shown
- Such a client allows client systems 1110 , 1120 and 1130 to access data hosted by storage server 1140 A or 1140 B or one of storage devices 1160 A( 1 )-(N), 1160 B( 1 )-(N), 1180 ( 1 )-(N) or intelligent storage array 1190 .
- FIG. 11 depicts the use of a network such as the Internet for exchanging data, but the present invention is not limited to the Internet or any particular network-based environment.
- any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermediate components.
- any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality.
- signal bearing media include computer-readable storage media, transmission type media such as digital and analog communications links, as well as media storage and distribution systems developed in the future.
- the above-discussed embodiments can be implemented by software modules that perform one or more tasks associated with the embodiments.
- the software modules discussed herein may include script, batch, or other executable files.
- the software modules may be stored on a machine-readable or computer-readable storage media such as magnetic floppy disks, hard disks, semiconductor memory (e.g., RAM, ROM, and flash-type media), optical discs (e.g., CD-ROMs, CD-Rs, and DVDs), or other types of memory modules.
- a storage device used for storing firmware or hardware modules in accordance with an embodiment of the invention can also include a semiconductor-based memory, which may be permanently, removably or remotely coupled to a microprocessor/memory system.
- the modules can be stored within a computer system memory to configure the computer system to perform the functions of the module.
- Other new and various types of computer-readable storage media may be used to store the modules discussed herein.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/363,207 US8074043B1 (en) | 2009-01-30 | 2009-01-30 | Method and apparatus to recover from interrupted data streams in a deduplication system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/363,207 US8074043B1 (en) | 2009-01-30 | 2009-01-30 | Method and apparatus to recover from interrupted data streams in a deduplication system |
Publications (1)
Publication Number | Publication Date |
---|---|
US8074043B1 true US8074043B1 (en) | 2011-12-06 |
Family
ID=45034517
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/363,207 Active 2030-05-29 US8074043B1 (en) | 2009-01-30 | 2009-01-30 | Method and apparatus to recover from interrupted data streams in a deduplication system |
Country Status (1)
Country | Link |
---|---|
US (1) | US8074043B1 (en) |
Cited By (49)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100198797A1 (en) * | 2009-02-05 | 2010-08-05 | Wideman Roderick B | Classifying data for deduplication and storage |
US20120059800A1 (en) * | 2010-09-03 | 2012-03-08 | Fanglu Guo | System and method for scalable reference management in a deduplication based storage system |
US20120084269A1 (en) * | 2010-09-30 | 2012-04-05 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US20120117066A1 (en) * | 2010-08-20 | 2012-05-10 | Florian Goette | Computer implemented method for processing data on an internet-accessible data processing unit |
US20120271823A1 (en) * | 2011-04-25 | 2012-10-25 | Rovi Technologies Corporation | Automated discovery of content and metadata |
US8572163B1 (en) * | 2009-08-31 | 2013-10-29 | Symantec Corporation | Systems and methods for deduplicating data based on performance of a deduplication system |
US8572340B2 (en) | 2010-09-30 | 2013-10-29 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US20130339315A1 (en) * | 2012-06-15 | 2013-12-19 | Quantum Corporation | Configurable Data Generator |
US20140095672A1 (en) * | 2010-03-05 | 2014-04-03 | Samsung Electronics Co., Ltd. | Method and apparatus for generating a reproducing adaptive stream based on file format, and recording medium thereof |
US8930306B1 (en) | 2009-07-08 | 2015-01-06 | Commvault Systems, Inc. | Synchronized data deduplication |
US8954446B2 (en) | 2010-12-14 | 2015-02-10 | Comm Vault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US9020900B2 (en) | 2010-12-14 | 2015-04-28 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US20150234616A1 (en) * | 2009-06-25 | 2015-08-20 | Emc Corporation | System and method for providing long-term storage for data |
US9218375B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US9223840B2 (en) | 2012-12-31 | 2015-12-29 | Futurewei Technologies, Inc. | Fast object fingerprints |
US9405763B2 (en) | 2008-06-24 | 2016-08-02 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US9519587B2 (en) | 2013-06-21 | 2016-12-13 | Huawei Technologies Co., Ltd. | Pre-reading file containers storing unread file segments and segments that do not belong to the file |
US9575673B2 (en) | 2014-10-29 | 2017-02-21 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US9628535B2 (en) * | 2011-04-15 | 2017-04-18 | International Business Machines Corporation | Data streaming infrastructure for remote execution in a constrained environment |
US9633056B2 (en) | 2014-03-17 | 2017-04-25 | Commvault Systems, Inc. | Maintaining a deduplication database |
US9633033B2 (en) | 2013-01-11 | 2017-04-25 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US9794326B1 (en) * | 2014-06-20 | 2017-10-17 | Amazon Technologies, Inc. | Log information transmission integrity |
US9817599B2 (en) | 2015-05-11 | 2017-11-14 | Hewlett Packard Enterprise Development Lp | Storing indicators of unreferenced memory addresses in volatile memory |
US10061663B2 (en) | 2015-12-30 | 2018-08-28 | Commvault Systems, Inc. | Rebuilding deduplication data in a distributed deduplication data storage system |
US10078451B1 (en) * | 2016-01-22 | 2018-09-18 | Red Hat, Inc. | Deduplicating data based on boundary identification |
US10275397B2 (en) | 2013-02-22 | 2019-04-30 | Veritas Technologies Llc | Deduplication storage system with efficient reference updating and space reclamation |
US10339106B2 (en) | 2015-04-09 | 2019-07-02 | Commvault Systems, Inc. | Highly reusable deduplication database after disaster recovery |
US10380072B2 (en) | 2014-03-17 | 2019-08-13 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US10481826B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US20200241971A1 (en) * | 2019-01-28 | 2020-07-30 | EMC IP Holding Company LLC | Method and system to tag and route the striped backups to a single deduplication instance on a deduplication appliance |
US10877945B1 (en) * | 2013-12-20 | 2020-12-29 | EMC IP Holding Company LLC | Optimized block storage for change block tracking systems |
US10938765B1 (en) * | 2016-03-11 | 2021-03-02 | Veritas Technologies Llc | Systems and methods for preparing email databases for analysis |
US11010258B2 (en) | 2018-11-27 | 2021-05-18 | Commvault Systems, Inc. | Generating backup copies through interoperability between components of a data storage management system and appliances for data storage and deduplication |
US11249858B2 (en) | 2014-08-06 | 2022-02-15 | Commvault Systems, Inc. | Point-in-time backups of a production application made accessible over fibre channel and/or ISCSI as data sources to a remote application by representing the backups as pseudo-disks operating apart from the production application and its host |
US11277473B1 (en) * | 2020-12-01 | 2022-03-15 | Adp, Llc | Coordinating breaking changes in automatic data exchange |
US11294768B2 (en) | 2017-06-14 | 2022-04-05 | Commvault Systems, Inc. | Live browsing of backed up data residing on cloned disks |
US11314424B2 (en) | 2015-07-22 | 2022-04-26 | Commvault Systems, Inc. | Restore for block-level backups |
US11321195B2 (en) | 2017-02-27 | 2022-05-03 | Commvault Systems, Inc. | Hypervisor-independent reference copies of virtual machine payload data based on block-level pseudo-mount |
US11409448B2 (en) * | 2017-03-09 | 2022-08-09 | Netapp Inc. | Selectively storing data into allocation areas using streams |
US11416341B2 (en) | 2014-08-06 | 2022-08-16 | Commvault Systems, Inc. | Systems and methods to reduce application downtime during a restore operation using a pseudo-storage device |
US11436038B2 (en) | 2016-03-09 | 2022-09-06 | Commvault Systems, Inc. | Hypervisor-independent block-level live browse for access to backed up virtual machine (VM) data and hypervisor-free file-level recovery (block- level pseudo-mount) |
US11442896B2 (en) | 2019-12-04 | 2022-09-13 | Commvault Systems, Inc. | Systems and methods for optimizing restoration of deduplicated data stored in cloud-based storage resources |
US11463264B2 (en) | 2019-05-08 | 2022-10-04 | Commvault Systems, Inc. | Use of data block signatures for monitoring in an information management system |
US20230112207A1 (en) * | 2021-10-13 | 2023-04-13 | EMC IP Holding Company LLC | Stream optimized backup to cloud object store |
US11687424B2 (en) | 2020-05-28 | 2023-06-27 | Commvault Systems, Inc. | Automated media agent state management |
US11694094B2 (en) | 2018-03-21 | 2023-07-04 | Swim.IT Inc | Inferring digital twins from captured data |
US11698727B2 (en) | 2018-12-14 | 2023-07-11 | Commvault Systems, Inc. | Performing secondary copy operations based on deduplication performance |
US11829251B2 (en) | 2019-04-10 | 2023-11-28 | Commvault Systems, Inc. | Restore using deduplicated secondary copy data |
US20240020317A1 (en) * | 2022-07-12 | 2024-01-18 | Accenture Global Solutions Limited | Providing energy efficient dynamic redundancy elimination for stored data |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100082547A1 (en) * | 2008-09-22 | 2010-04-01 | Riverbed Technology, Inc. | Log Structured Content Addressable Deduplicating Storage |
US20110022825A1 (en) * | 2009-07-21 | 2011-01-27 | Spackman Stephen Philip | Creating and managing links to deduplication information |
-
2009
- 2009-01-30 US US12/363,207 patent/US8074043B1/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100082529A1 (en) * | 2008-05-08 | 2010-04-01 | Riverbed Technology, Inc. | Log Structured Content Addressable Deduplicating Storage |
US20100082547A1 (en) * | 2008-09-22 | 2010-04-01 | Riverbed Technology, Inc. | Log Structured Content Addressable Deduplicating Storage |
US20110022825A1 (en) * | 2009-07-21 | 2011-01-27 | Spackman Stephen Philip | Creating and managing links to deduplication information |
Cited By (111)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9405763B2 (en) | 2008-06-24 | 2016-08-02 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US11016859B2 (en) | 2008-06-24 | 2021-05-25 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US9176978B2 (en) * | 2009-02-05 | 2015-11-03 | Roderick B. Wideman | Classifying data for deduplication and storage |
US20100198797A1 (en) * | 2009-02-05 | 2010-08-05 | Wideman Roderick B | Classifying data for deduplication and storage |
US20150234616A1 (en) * | 2009-06-25 | 2015-08-20 | Emc Corporation | System and method for providing long-term storage for data |
US10108353B2 (en) * | 2009-06-25 | 2018-10-23 | EMC IP Holding Company LLC | System and method for providing long-term storage for data |
US11288235B2 (en) | 2009-07-08 | 2022-03-29 | Commvault Systems, Inc. | Synchronized data deduplication |
US10540327B2 (en) | 2009-07-08 | 2020-01-21 | Commvault Systems, Inc. | Synchronized data deduplication |
US8930306B1 (en) | 2009-07-08 | 2015-01-06 | Commvault Systems, Inc. | Synchronized data deduplication |
US8572163B1 (en) * | 2009-08-31 | 2013-10-29 | Symantec Corporation | Systems and methods for deduplicating data based on performance of a deduplication system |
US10630759B2 (en) * | 2010-03-05 | 2020-04-21 | Samsung Electronics Co., Ltd | Method and apparatus for generating and reproducing adaptive stream based on file format, and recording medium thereof |
US9906580B2 (en) * | 2010-03-05 | 2018-02-27 | Samsung Electronics Co., Ltd | Method and apparatus for generating and reproducing adaptive stream based on file format, and recording medium thereof |
US20140095672A1 (en) * | 2010-03-05 | 2014-04-03 | Samsung Electronics Co., Ltd. | Method and apparatus for generating a reproducing adaptive stream based on file format, and recording medium thereof |
US20120117066A1 (en) * | 2010-08-20 | 2012-05-10 | Florian Goette | Computer implemented method for processing data on an internet-accessible data processing unit |
US8782011B2 (en) | 2010-09-03 | 2014-07-15 | Symantec Corporation | System and method for scalable reference management in a deduplication based storage system |
US8392376B2 (en) * | 2010-09-03 | 2013-03-05 | Symantec Corporation | System and method for scalable reference management in a deduplication based storage system |
US20120059800A1 (en) * | 2010-09-03 | 2012-03-08 | Fanglu Guo | System and method for scalable reference management in a deduplication based storage system |
US9239687B2 (en) | 2010-09-30 | 2016-01-19 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US8572340B2 (en) | 2010-09-30 | 2013-10-29 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US9110602B2 (en) | 2010-09-30 | 2015-08-18 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US20120084269A1 (en) * | 2010-09-30 | 2012-04-05 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US10126973B2 (en) | 2010-09-30 | 2018-11-13 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US8364652B2 (en) | 2010-09-30 | 2013-01-29 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US8578109B2 (en) | 2010-09-30 | 2013-11-05 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US8577851B2 (en) * | 2010-09-30 | 2013-11-05 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US9898225B2 (en) | 2010-09-30 | 2018-02-20 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US9639289B2 (en) | 2010-09-30 | 2017-05-02 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US9619480B2 (en) | 2010-09-30 | 2017-04-11 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US9116850B2 (en) | 2010-12-14 | 2015-08-25 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US9104623B2 (en) | 2010-12-14 | 2015-08-11 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US9898478B2 (en) | 2010-12-14 | 2018-02-20 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US11422976B2 (en) | 2010-12-14 | 2022-08-23 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US9020900B2 (en) | 2010-12-14 | 2015-04-28 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US10740295B2 (en) | 2010-12-14 | 2020-08-11 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US10191816B2 (en) | 2010-12-14 | 2019-01-29 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US8954446B2 (en) | 2010-12-14 | 2015-02-10 | Comm Vault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US11169888B2 (en) | 2010-12-14 | 2021-11-09 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US9628535B2 (en) * | 2011-04-15 | 2017-04-18 | International Business Machines Corporation | Data streaming infrastructure for remote execution in a constrained environment |
US20120271823A1 (en) * | 2011-04-25 | 2012-10-25 | Rovi Technologies Corporation | Automated discovery of content and metadata |
US10176053B2 (en) | 2012-06-13 | 2019-01-08 | Commvault Systems, Inc. | Collaborative restore in a networked storage system |
US9218374B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Collaborative restore in a networked storage system |
US9218376B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Intelligent data sourcing in a networked storage system |
US10387269B2 (en) | 2012-06-13 | 2019-08-20 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US9218375B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US10956275B2 (en) | 2012-06-13 | 2021-03-23 | Commvault Systems, Inc. | Collaborative restore in a networked storage system |
US9858156B2 (en) | 2012-06-13 | 2018-01-02 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US9251186B2 (en) | 2012-06-13 | 2016-02-02 | Commvault Systems, Inc. | Backup using a client-side signature repository in a networked storage system |
US20130339315A1 (en) * | 2012-06-15 | 2013-12-19 | Quantum Corporation | Configurable Data Generator |
US8983916B2 (en) * | 2012-06-15 | 2015-03-17 | Timothy STOAKES | Configurable data generator |
US9223840B2 (en) | 2012-12-31 | 2015-12-29 | Futurewei Technologies, Inc. | Fast object fingerprints |
US10229133B2 (en) | 2013-01-11 | 2019-03-12 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US9665591B2 (en) | 2013-01-11 | 2017-05-30 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US9633033B2 (en) | 2013-01-11 | 2017-04-25 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US11157450B2 (en) | 2013-01-11 | 2021-10-26 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US10275397B2 (en) | 2013-02-22 | 2019-04-30 | Veritas Technologies Llc | Deduplication storage system with efficient reference updating and space reclamation |
US9519587B2 (en) | 2013-06-21 | 2016-12-13 | Huawei Technologies Co., Ltd. | Pre-reading file containers storing unread file segments and segments that do not belong to the file |
US10877945B1 (en) * | 2013-12-20 | 2020-12-29 | EMC IP Holding Company LLC | Optimized block storage for change block tracking systems |
US11119984B2 (en) | 2014-03-17 | 2021-09-14 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US9633056B2 (en) | 2014-03-17 | 2017-04-25 | Commvault Systems, Inc. | Maintaining a deduplication database |
US10445293B2 (en) | 2014-03-17 | 2019-10-15 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US11188504B2 (en) | 2014-03-17 | 2021-11-30 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US10380072B2 (en) | 2014-03-17 | 2019-08-13 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US9794326B1 (en) * | 2014-06-20 | 2017-10-17 | Amazon Technologies, Inc. | Log information transmission integrity |
US10511656B1 (en) | 2014-06-20 | 2019-12-17 | Amazon Technologies, Inc. | Log information transmission integrity |
US10122789B1 (en) | 2014-06-20 | 2018-11-06 | Amazon Technologies, Inc. | Log information transmission integrity |
US11249858B2 (en) | 2014-08-06 | 2022-02-15 | Commvault Systems, Inc. | Point-in-time backups of a production application made accessible over fibre channel and/or ISCSI as data sources to a remote application by representing the backups as pseudo-disks operating apart from the production application and its host |
US11416341B2 (en) | 2014-08-06 | 2022-08-16 | Commvault Systems, Inc. | Systems and methods to reduce application downtime during a restore operation using a pseudo-storage device |
US9575673B2 (en) | 2014-10-29 | 2017-02-21 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US11921675B2 (en) | 2014-10-29 | 2024-03-05 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US11113246B2 (en) | 2014-10-29 | 2021-09-07 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US10474638B2 (en) | 2014-10-29 | 2019-11-12 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US9934238B2 (en) | 2014-10-29 | 2018-04-03 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US10339106B2 (en) | 2015-04-09 | 2019-07-02 | Commvault Systems, Inc. | Highly reusable deduplication database after disaster recovery |
US11301420B2 (en) | 2015-04-09 | 2022-04-12 | Commvault Systems, Inc. | Highly reusable deduplication database after disaster recovery |
US9817599B2 (en) | 2015-05-11 | 2017-11-14 | Hewlett Packard Enterprise Development Lp | Storing indicators of unreferenced memory addresses in volatile memory |
US10481826B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US10481824B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US10481825B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US11733877B2 (en) | 2015-07-22 | 2023-08-22 | Commvault Systems, Inc. | Restore for block-level backups |
US11314424B2 (en) | 2015-07-22 | 2022-04-26 | Commvault Systems, Inc. | Restore for block-level backups |
US10877856B2 (en) | 2015-12-30 | 2020-12-29 | Commvault Systems, Inc. | System for redirecting requests after a secondary storage computing device failure |
US10061663B2 (en) | 2015-12-30 | 2018-08-28 | Commvault Systems, Inc. | Rebuilding deduplication data in a distributed deduplication data storage system |
US10956286B2 (en) | 2015-12-30 | 2021-03-23 | Commvault Systems, Inc. | Deduplication replication in a distributed deduplication data storage system |
US10255143B2 (en) | 2015-12-30 | 2019-04-09 | Commvault Systems, Inc. | Deduplication replication in a distributed deduplication data storage system |
US10310953B2 (en) | 2015-12-30 | 2019-06-04 | Commvault Systems, Inc. | System for redirecting requests after a secondary storage computing device failure |
US10592357B2 (en) | 2015-12-30 | 2020-03-17 | Commvault Systems, Inc. | Distributed file system in a distributed deduplication data storage system |
US10795578B2 (en) | 2016-01-22 | 2020-10-06 | Red Hat, Inc. | Deduplicating data based on boundary identification |
US10078451B1 (en) * | 2016-01-22 | 2018-09-18 | Red Hat, Inc. | Deduplicating data based on boundary identification |
US11436038B2 (en) | 2016-03-09 | 2022-09-06 | Commvault Systems, Inc. | Hypervisor-independent block-level live browse for access to backed up virtual machine (VM) data and hypervisor-free file-level recovery (block- level pseudo-mount) |
US10938765B1 (en) * | 2016-03-11 | 2021-03-02 | Veritas Technologies Llc | Systems and methods for preparing email databases for analysis |
US11321195B2 (en) | 2017-02-27 | 2022-05-03 | Commvault Systems, Inc. | Hypervisor-independent reference copies of virtual machine payload data based on block-level pseudo-mount |
US12001301B2 (en) | 2017-02-27 | 2024-06-04 | Commvault Systems, Inc. | Hypervisor-independent reference copies of virtual machine payload data based on block-level pseudo-mount |
US11409448B2 (en) * | 2017-03-09 | 2022-08-09 | Netapp Inc. | Selectively storing data into allocation areas using streams |
US11294768B2 (en) | 2017-06-14 | 2022-04-05 | Commvault Systems, Inc. | Live browsing of backed up data residing on cloned disks |
US11694094B2 (en) | 2018-03-21 | 2023-07-04 | Swim.IT Inc | Inferring digital twins from captured data |
US11681587B2 (en) | 2018-11-27 | 2023-06-20 | Commvault Systems, Inc. | Generating copies through interoperability between a data storage management system and appliances for data storage and deduplication |
US11010258B2 (en) | 2018-11-27 | 2021-05-18 | Commvault Systems, Inc. | Generating backup copies through interoperability between components of a data storage management system and appliances for data storage and deduplication |
US11698727B2 (en) | 2018-12-14 | 2023-07-11 | Commvault Systems, Inc. | Performing secondary copy operations based on deduplication performance |
EP3690655A1 (en) * | 2019-01-28 | 2020-08-05 | EMC IP Holding Company LLC | Method and system to tag and route the striped backups to a single deduplication instance on a deduplication appliance |
CN111488242A (en) * | 2019-01-28 | 2020-08-04 | Emc知识产权控股有限公司 | Method and system for tagging and routing a striped backup to a single deduplication instance on a deduplication appliance |
US20200241971A1 (en) * | 2019-01-28 | 2020-07-30 | EMC IP Holding Company LLC | Method and system to tag and route the striped backups to a single deduplication instance on a deduplication appliance |
US10922188B2 (en) * | 2019-01-28 | 2021-02-16 | EMC IP Holding Company LLC | Method and system to tag and route the striped backups to a single deduplication instance on a deduplication appliance |
US11829251B2 (en) | 2019-04-10 | 2023-11-28 | Commvault Systems, Inc. | Restore using deduplicated secondary copy data |
US11463264B2 (en) | 2019-05-08 | 2022-10-04 | Commvault Systems, Inc. | Use of data block signatures for monitoring in an information management system |
US11442896B2 (en) | 2019-12-04 | 2022-09-13 | Commvault Systems, Inc. | Systems and methods for optimizing restoration of deduplicated data stored in cloud-based storage resources |
US11687424B2 (en) | 2020-05-28 | 2023-06-27 | Commvault Systems, Inc. | Automated media agent state management |
US11277473B1 (en) * | 2020-12-01 | 2022-03-15 | Adp, Llc | Coordinating breaking changes in automatic data exchange |
US20230112207A1 (en) * | 2021-10-13 | 2023-04-13 | EMC IP Holding Company LLC | Stream optimized backup to cloud object store |
US11874747B2 (en) * | 2021-10-13 | 2024-01-16 | EMC IP Holding Company LLC | Stream optimized backup to cloud object store |
US20240020317A1 (en) * | 2022-07-12 | 2024-01-18 | Accenture Global Solutions Limited | Providing energy efficient dynamic redundancy elimination for stored data |
US11989210B2 (en) * | 2022-07-12 | 2024-05-21 | Accenture Global Solutions Limited | Providing energy efficient dynamic redundancy elimination for stored data |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8074043B1 (en) | Method and apparatus to recover from interrupted data streams in a deduplication system | |
US8572055B1 (en) | Method and system for efficiently handling small files in a single instance storage data store | |
US8204868B1 (en) | Method and system for improving performance with single-instance-storage volumes by leveraging data locality | |
US8650371B1 (en) | Method and system for efficient space management for single-instance-storage volumes | |
US9298707B1 (en) | Efficient data storage and retrieval for backup systems | |
US9928210B1 (en) | Constrained backup image defragmentation optimization within deduplication system | |
US9558075B2 (en) | Synthetic full backup generation | |
US7769949B1 (en) | Method and apparatus for providing a temporal storage appliance with block virtualization in storage networks | |
US20120036106A1 (en) | Data Replication Techniques Using Incremental Checkpoints | |
US7523277B1 (en) | Transient point-in-time images for continuous data protection | |
US8904137B1 (en) | Deduplication system space recycling through inode manipulation | |
JP4473694B2 (en) | Long-term data protection system and method | |
US8555278B2 (en) | Method and system for migrating a selected set of virtual machines between volumes | |
US9176853B2 (en) | Managing copy-on-writes to snapshots | |
US9075532B1 (en) | Self-referential deduplication | |
US10296428B2 (en) | Continuous replication in a distributed computer system environment | |
US8341119B1 (en) | Flexible copies having different sub-types | |
US10922009B2 (en) | Mirroring write operations across data storage devices | |
US8694745B2 (en) | Physical to virtual disks creation (P2V) method, by harvesting data from critical sectors | |
US8392437B2 (en) | Method and system for providing deduplication information to applications | |
US11366724B2 (en) | Low cost, heterogeneous method of transforming replicated data for consumption in the cloud | |
US9268811B1 (en) | Replay of writes in replication log | |
US20170161150A1 (en) | Method and system for efficient replication of files using shared null mappings when having trim operations on files | |
US8412824B1 (en) | Systems and methods for dynamically managing the migration of a single instance of data between storage devices | |
US11360699B1 (en) | Method and system for improved write performance in erasure-coded storage systems |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SYMANTEC CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ZEIS, MICHAEL JOHN;REEL/FRAME:022183/0128 Effective date: 20090130 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: VERITAS US IP HOLDINGS LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SYMANTEC CORPORATION;REEL/FRAME:037697/0412 Effective date: 20160129 |
|
AS | Assignment |
Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH CAROLINA Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0001 Effective date: 20160129 Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT, CONNECTICUT Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0726 Effective date: 20160129 Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0001 Effective date: 20160129 Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATE Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0726 Effective date: 20160129 |
|
AS | Assignment |
Owner name: VERITAS TECHNOLOGIES LLC, CALIFORNIA Free format text: MERGER AND CHANGE OF NAME;ASSIGNORS:VERITAS US IP HOLDINGS LLC;VERITAS TECHNOLOGIES LLC;REEL/FRAME:038455/0752 Effective date: 20160329 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
AS | Assignment |
Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT, DELAWARE Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS TECHNOLOGIES LLC;REEL/FRAME:054370/0134 Effective date: 20200820 |
|
AS | Assignment |
Owner name: VERITAS US IP HOLDINGS, LLC, CALIFORNIA Free format text: TERMINATION AND RELEASE OF SECURITY IN PATENTS AT R/F 037891/0726;ASSIGNOR:WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT;REEL/FRAME:054535/0814 Effective date: 20201127 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |