JP2011510405A - Scalable deduplication mechanism - Google Patents

Abstract

  A method is provided for removing redundant data from a backup storage system. In one example, the method receives an application layer data object and selects a deduplication domain from a plurality of deduplication domains based at least in part on data object characteristics associated with the deduplication domain. And determining that the application layer data object has the characteristics and directing the application layer data object to the deduplication domain.

Description

Background 1. FIELD OF THE INVENTION Aspects of the invention relate to data storage, and more particularly to an apparatus and method for providing an extensible data deduplication service.

2. Description of Related Art Many computer systems include one or more host computers and one or more data storage systems that store data used by the host computers. These host computers and storage systems are typically networked together using a network, such as a Fiber Channel network, an Ethernet network, or another type of communication network. Fiber Channel is a standard that combines the speed of a channel-based transmission scheme with the flexibility of a network-based transmission scheme to allow multiple initiators to communicate with multiple targets over a network. And the target can be any device coupled to the network. Since Fiber Channel is typically implemented using a high-speed transmission medium such as a fiber optic cable, it is a common choice for storage system networks where large amounts of data are transferred.

  An example of a typical networked computing environment including several host computers and a backup storage system is shown in FIG. One or more application servers 102 are coupled to a plurality of user computers 104 via a local area network (LAN) 103. Both application server 102 and user computer 104 may be considered “host computers”. Application server 102 is coupled to one or more primary storage devices 106 via a storage area network (SAN) 108. The primary storage device 106 may be a disk array that can be obtained from companies such as EMC and IBM. Alternatively, a bus (not shown) or other network link may provide an interconnection between the application server and the primary storage system 106. The bus and / or Fiber Channel network connection may be a Small Component System Interconnect (SCSI) protocol that dictates the format of the packets transferred between the host computer (eg, application server 102) and the storage system 106. It may operate using the following protocol.

  It should be appreciated that the networked computing environment shown in FIG. 1 is specific to large systems such as those used by large financial institutions or large corporations. It should be understood that many networked computing environments need not include all of the elements shown in FIG. For example, a smaller networked computing environment may simply include a host computer connected directly or via a LAN to a storage system. In addition, although FIG. 1 illustrates a separate user computer 104, application server 102, and media server 114, these functions may be combined in one or more computers.

  Many networked computing environments include at least one secondary or backup storage system 110 in addition to the primary storage device 106. The backup storage system 110 may typically be a tape library, but other secondary storage systems with large capacity and reliability may be used. These secondary storage systems are typically slower than primary storage devices, but include some type of removable media (eg, tape, magnetic disk or optical disk) that can be removed and stored off-site.

  In the illustrated example, application server 102 may be able to communicate directly with backup storage system 110 via, for example, Ethernet or other communication link 112. However, such connections are relatively slow and may use up resources such as processor time or network bandwidth. Thus, a system such as that shown may include one or more media servers 114 that may provide a communication link 115 between the SAN 108 and the backup storage system 110 using, for example, Fiber Channel.

  Media server 114 provides a backup / restore application that controls the transfer of data between the host computer (such as user computer 104, media server 114, and / or application server 102), primary storage device 106, and backup storage system 110. Software may be executed. Examples of backup / restore applications are available from companies such as Veritas and Legato. For data protection, as is known in the art, data from various host computers and / or primary storage devices in a networked computing environment is periodically backed up using a backup / restore application. 110 may be backed up.

  Of course, as noted above, it should be recognized that many networked computer environments are smaller and may have fewer components than the exemplary networked computer environment shown in FIG. Thus, the media server 114 may actually be combined with the application server 102 on a single host computer, and the backup / restore application is coupled to the backup storage system 110 (directly or indirectly, such as through a network). It should also be appreciated that it can be executed on any host computer that has been designated.

  An example of a typical backup storage system is a tape library that includes a number of tape cartridges, at least one tape drive, and a robotic mechanism that controls the attachment and removal of cartridges from and to the tape drive. The backup / restore application directs the robotic mechanism to locate a particular tape cartridge, eg, tape number 0001, and attach the tape cartridge to the tape drive so that data is written to the tape. The backup / restore application also controls the format in which data is written to tape. Typically, a backup / restore application uses SCSI commands or other standardized commands to instruct the robotics to control the tape drive to write data to the tape, and to write previously written data. You may recover from the tape.

  Conventional tape library backup systems have a number of problems including speed, reliability, and capacity locking. Many large companies need to back up terabytes of data weekly. However, expensive and high performance tapes can usually only read / write data at a rate of 30 to 40 megabytes per second (MB / s), ie about 50 gigabytes per hour (GB / hr). For this reason, it may take at least 10 to 20 hours of continuous data transfer time to back up 1 or 2 terabytes of data to the tape backup system.

  In addition, many tape manufacturers have dropped tapes (as can happen relatively frequently in a normal tape library, as human operators or robotic mechanisms may drop tapes during movement or mounting operations). Or if the tape is exposed to non-ideal environmental conditions such as extreme temperatures or humidity, it will not guarantee that the data can be stored on (or restored from) the tape. . Therefore, great care must be taken to store the tape in a controlled environment. In addition, the complex machinery of tape libraries (including robotic mechanisms) is expensive to maintain, and individual tape cartridges are relatively expensive and have a limited service life.

  In view of the costs associated with traditional tape libraries and other types of backup storage media, suppliers often incorporate deduplication processing into their product offerings to reduce overall backup media requirements. Deduplication is the process of identifying a repeating sequence of data over time, ie it is a manifestation of delta compression. Deduplication is typically implemented as a function of a target device such as a backup storage device. The act of identifying redundant data in a backup data stream is complex and has been solved in the current state of the art using hash fingerprinting and pattern recognition.

  In hash fingerprinting, the received data stream is first subjected to an alignment process (attempts to predict a good “breakpoint”, also known as an edge, in the data stream that provides the highest likelihood of subsequent matches). Next, it undergoes hash processing (usually SHA-1 in the current state of the art). The data stream is broken into several chunks (typically about 8 to 12 kilobytes in size) by hashing, and each chunk is assigned its resulting hash value. This hash value is compared with the memory resident table. If a hash entry is found, the data is assumed to be redundant and is replaced with a pointer to an existing data block already stored in the disk storage system. The position of existing data is listed on the table. If no hash entry is found, the data is stored in the disk storage system and its location is recorded in a memory resident table along with the hash. Some examples illustrating this mechanism include US Pat. No. 7,065,619 assigned to Data Domain and US Pat. No. 5,990,810 assigned to Quantum Corporation. Can be found in the issue. Hash fingerprinting is typically performed inline. That is, the data is processed in real time before being written to disk.

  According to pattern recognition, the received data stream is first “lumped” or split into relatively large data blocks (approximately 32 MB). The data is then processed by a simple rolling hash method that builds a list of hash values. A transformation is performed on the hash value, where the resulting small list of values represents the “fingerprint” of the data block. Next, a search is performed against the hash table to find at least some number of fingerprint hashes present in other given storage blocks. If the minimum number of matches is not met, the block is considered unique and is stored directly on disk. The corresponding fingerprint hash is added to the memory resident table. If the minimum number of matches is met, the current data block may match a previously stored data block. In this case, the disk storage block allocated by the matching fingerprint is read into memory and compared byte by byte with the previously hashed candidate block. If the entire sequence of data is equal, the data block is replaced with a pointer to a physically addressed block of storage. If the entire block does not match, a delta difference mechanism is employed to determine the smallest data set in the block that needs to be stored. The result is a combination of unique data and a reference to a closely matching block of previously stored data. One example illustrating this mechanism can be found in US Patent Application No. US 2006/0059207, assigned to Diligent Corporation. As described above, this operation is typically performed inline.

SUMMARY OF THE INVENTION Aspects and embodiments of the present invention overcome or alleviate some or all of the problems of conventional data deduplication techniques and are more effective and extended than data storage systems that incorporate traditional deduplication techniques. A data storage system is provided that can provide functionality.

  In overview, aspects and embodiments of the present invention provide a random access-based emulation that emulates a conventional tape backup storage system so that a backup / restore application sees the same display of devices and media as a physical tape library. Provide a storage system. The storage system of the present invention uses software and hardware to emulate physical tape media and replace them with one or more random access disk arrays to convert linear sequential data in tape format to disk. Convert to data suitable for storage.

  In accordance with some aspects and embodiments of the present invention, for decrypting an existing backup data set and storing metadata (ie, data representing information about user data) in a searchable metadata cache Mechanisms, mechanisms that allow searching and / or browsing the metadata cache for files or objects, and these files or objects from data stored through existing backup policies and practices of typical backup software A mechanism for downloading via a web connection. Further, a mechanism for authenticating the user through an existing authentication mechanism and restricting the display of the metadata cache based on the current user authentication information may be included.

  Aspects and embodiments of the present invention also provide removal of redundant data from backup data objects. This removal process, which can be referred to as “deduplication”, reduces the storage capacity required to hold a copy of the back data and thus the amount of electronic media required to store the backup data. Embodiments of a deduplication process in accordance with at least some aspects of the present invention efficiently use computing resources and optimize the deduplication process by using metadata.

  As described further below, some embodiments are directed to intelligent guidance of the entire deduplication process. In some of these embodiments, the data storage system uses software and hardware to direct the data object to one of several deduplication domains for deduplication and storage. In addition, hardware and / or software implemented applications are provided to configure deduplication domains that manage data deduplication within the constraints presented by a given data storage system. Some embodiments assert the recognition that traditional hash fingerprinting techniques are constrained by the amount of memory available. Other embodiments reflect the recognition that with the pattern recognition approach, random I / O workload is a substantial limitation. Thus, these examples demonstrate the recognition of restrictions imposed by conventional hash fingerprinting and pattern recognition de-duplication techniques.

  In accordance with other aspects and embodiments of the present invention, a mechanism for logically merging multiple cartridge representations in a metadata cache, and appropriately assigning labels and barcodes to newly synthesized cartridges, including: And a mechanism for being accepted into the backup / restore software as a valid data set. In accordance with yet another aspect and embodiment of the present invention, to store multiple copies of a data element representing a composite cartridge, or to store only pointers to existing data represented in a metadata cache A mechanism is provided.

  According to one embodiment, a method is provided for inducing deduplication of application layer data objects. The method includes an act of receiving the application layer data object, an act of selecting a deduplication domain from a plurality of deduplication domains based at least in part on data object characteristics associated with the deduplication domain; An act of determining that the application layer data object has the characteristic and an act of inducing the application layer data object to the selected deduplication domain.

  In one example, the act of receiving the application layer data object may include an act of receiving a data stream and an act of identifying the application layer data object using metadata included in the data stream. In another example, the act of receiving the data stream may include the act of receiving the multiplexed data stream. According to another example, the method may further include an act of extracting metadata included in the data stream using the application layer data object. In yet another example, the act of selecting the deduplication domain from the plurality of deduplication domains includes: extracting the extracted metadata associated with the application layer data object with at least one associated with the deduplication domain. It may include an act of comparing two such characteristics. According to yet another example, the act of extracting the metadata included in the data stream includes backup policy name, data source type, data source name, backup application name, operation system type, data type, backup An act of extracting at least one of type, file name, directory structure, and time series information may be included.

  In another example, the method may further include an act of configuring each of the plurality of deduplication domains to use one of a plurality of deduplication methods. According to another example, the act of configuring each of the plurality of deduplication domains uses a deduplication method selected from the group comprising hash fingerprinting, pattern recognition, and content recognition deduplication, The act which comprises each of the said several deduplication domain may be included. In yet another example, the method may further include an act of associating each of the plurality of deduplication domains with at least one data object characteristic. According to additional examples, the method can include deduplicating the application layer data object within the selected deduplication domain and the plurality of deduplication domains based on a result of the deduplication action. And further adjusting the data object characteristic associated with at least one of the two. In yet another example, the act of adjusting the data object characteristics may include the act of storing data in a deduplication domain database.

  According to another embodiment, a grid computing environment is provided for performing the acts of the method for directing deduplication of application layer data objects as described above.

  According to another embodiment, a backup storage system is provided to perform the acts of the method for inducing deduplication of application layer data objects as described above. In this embodiment, the method is performed while data is not backed up to a backup storage system.

  According to another embodiment, a backup storage system is provided to perform the acts of the method for inducing deduplication of application layer data objects as described above. In this embodiment, the method is performed while data is being backed up to a backup storage system.

  According to another embodiment, a computer readable medium having stored thereon computer readable signals defining instructions is provided. These instructions are executed by the computer as a result of the act of receiving the application layer data object and a deduplication domain from a plurality of deduplication domains at least partially in the data object characteristics associated with the deduplication domain. Commanding the computer to perform the act of selecting based on the application layer, the act of determining that the application layer data object has the characteristic, and the act of inducing the application layer data object to the selected deduplication domain To do.

  According to another embodiment, a system is provided for inducing deduplication of application layer data objects. The system includes a plurality of deduplication domains, each of the deduplication domains being associated with at least one characteristic common to the plurality of application layer data objects, the system further comprising: Includes a controller coupled to the plurality of deduplication domains, the controller receiving the application layer data object, wherein the application layer data object has the at least one characteristic associated with a deduplication domain. In this case, the application layer data object is guided to the deduplication domain.

  In one example, the controller may be further configured to receive the data stream and identify the application layer data object using metadata included in the data stream. In another example, the data stream may be multiplexed. In another example, the controller may be further configured to extract metadata included in the data stream using the application layer data object. In yet another example, the controller further indicates that the application layer data object has the at least one characteristic associated with the deduplication domain, the extracted meta data associated with the application layer data object. The data may be configured to be determined by comparing the at least one characteristic associated with the deduplication domain. In yet another example, the controller further includes backup policy name, data source type, data source name, backup application name, operating system type, data type, backup type, file name, directory structure, and time series information. May be configured to extract at least one of them. In an additional example, the controller may be further configured to configure each of the plurality of deduplication domains to use one of a plurality of deduplication methods. In yet another example, the controller further configures each of the plurality of deduplication domains to use a deduplication method selected from the group comprising hash fingerprinting, pattern recognition, and content recognition deduplication. It may be configured to.

  According to another example, the controller may be further configured to associate each of the plurality of deduplication domains with at least one data object characteristic. In another example, the controller further causes deduplication of the application layer data object within the selected deduplication domain and, based on a result of the deduplication action, of the plurality of deduplication domains. It may be configured to adjust the data object characteristic associated with at least one. In yet another example, the controller may be further configured to store data in a deduplication domain database. In another example, the system may be included in a grid computing environment. In yet another example, the controller further receives the application layer data object while the data is not backed up to the system, and the application layer data object is associated with the at least one deduplication domain. The application layer data object may be configured to be guided to the deduplication domain by determining that the application layer data object has the characteristic. In addition, according to an example, the controller further receives the application layer data object while the data is backed up to the system and the at least the application layer data object associated with a deduplication domain is located. It may be configured to determine that it has one characteristic and direct the application layer data object to the deduplication domain.

  Further aspects, embodiments, and advantages of these exemplary aspects and embodiments are described in detail below. Additionally, both the foregoing information and the following detailed description are merely exemplary of various aspects and embodiments, and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and embodiments. Is intended. The accompanying drawings are included and incorporated in and constitute a part of this specification to provide an illustration and various understanding of the various aspects and examples. The drawings, together with the remainder of the specification, serve to explain the principles and operations of the described and claimed aspects and embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS Various aspects of at least one embodiment are described below with reference to the accompanying drawings. In the drawings that are not intended to be drawn to scale, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For clarity, not all components are named in all figures. The figures are provided for purposes of illustration and description and are not intended as a definition of the limits of the invention.

1 is a block diagram of an example large-scale networked computing environment that includes a backup storage system. FIG. 1 is a block diagram of an example networked computing environment including a storage system in accordance with aspects of the invention. FIG. 1 is a block diagram of an example of a storage system according to an aspect of the present invention. It is a block diagram showing a virtual layout of an example of a storage system according to an aspect of the present invention. 2 is a schematic layout of an example system file according to an aspect of the invention. 2 is an example of a tape directory structure according to an aspect of the invention. It is a figure which shows an example of the method of producing the synthetic full backup according to the situation of this invention. FIG. 2 is a schematic diagram of an example of a series of backup data sets including a synthetic full backup according to the present invention. It is a figure of an example of a metadata cache structure. It is a figure of an example of the virtual cartridge which memorize | stores a synthetic | combination full backup data set. FIG. 4 is a diagram of another example of a virtual cartridge that stores a synthetic full backup data set. 4 is a flowchart of a method for deduplicating data objects according to the present invention. FIG. 3 is a diagram of two backup data objects. FIG. 13B is a diagram of a deduplicated copy of the backup data object shown in FIG. 13A. FIG. 13B is another diagram of a deduplicated copy of the backup data object shown in FIG. 13A. FIG. 6 is a block diagram of an example of a deduplication inductor according to an aspect of the invention. 4 is a flowchart of a method for inducing deduplication of data objects according to the present invention.

DETAILED DESCRIPTION Various embodiments and their aspects will now be described in more detail with reference to the accompanying drawings. It should be appreciated that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced or carried out in various ways. Particular implementations are provided herein for illustrative purposes only and are not intended to be limiting. In particular, acts, elements, and features described in connection with any one or more embodiments are not intended to be excluded from a similar role in any other embodiments. Also, the expressions and terms used herein are for the purpose of description and should not be considered limiting. The use of “including”, “comprising”, “having”, “containing”, “involved”, and variations thereof herein includes the items listed below and their equivalents, as well as additional items. It is intended to include.

  Any of the embodiments disclosed herein may be combined with other embodiments, such as “one embodiment”, “some embodiments”, “alternative embodiments”, “various embodiments”, References to “one embodiment”, “at least one embodiment”, “this and other embodiments” are not necessarily mutually exclusive, and certain features described in connection with the embodiment It is intended to indicate that a structure, or characteristic may be included in at least one embodiment. The terms as used herein do not necessarily all refer to the same embodiment. Any embodiment may be combined with other embodiments in a manner consistent with the aspects disclosed herein. Reference to “or” is to be construed as inclusive, such that a term described using “or” indicates any one, two, or all of the described terms. obtain.

  As used herein, the term “host computer” refers to personal computers, workstations, mainframes, networked clients, servers, etc. that can communicate with other devices such as storage systems or other host computers. Any computer having at least one processor. The host computer may include a media server and application server (as described above with reference to FIG. 1) and a user computer (which may be a user workstation, PC, mainframe, etc.). In addition, in this disclosure, the term “networked computing environment” refers to a plurality of host computers connected to one or more shared storage systems in such a manner that the storage systems can communicate with each of the host computers. Including any computing environment. Fiber Channel is an example of a communication network that can be used in embodiments of the present invention. However, the network described herein is not limited to Fiber Channel, and instead of or in addition to Fiber Channel, various network components can be used in networks such as Token Ring or Ethernet. It should be appreciated that they can communicate with each other over a connection or through a combination of different network connections. Aspects of the invention can also be used in bus topologies such as SCSI or parallel SCSI.

  In accordance with various embodiments and aspects of the present invention, a virtual removable media library backup storage system is provided that can emulate a removable media based storage system using one or more disk arrays. With embodiments of the present invention, data can be transferred to removable media (tapes, magnetic disks, optical disks, etc.) without the need for users to modify or adjust existing backup procedures or purchase new backup / restore applications. Data can be backed up to a disk array using the same backup / restore application that has been used to back up. In one embodiment described in detail herein, the emulated removable media is a tape, and the backup storage system of the present invention includes a robotic mechanism used to handle tapes and tapes in a conventional tape library system. Emulate a tape library system that includes

  Data that can be backed up and restored using embodiments of the present invention can be organized into various data objects. These data objects may include any structure that can store data. An exemplary non-limiting list of data objects includes bits, bytes, data files, data blocks, data directories, backup data sets, and virtual cartridges, which are further described below. Although most of this disclosure refers to backup and restore of data files, embodiments of the invention may operate on any data object, and the term “data file” is interchangeable with “data object” Should be recognized. In addition, as will be appreciated by those skilled in the art, the embodiments described herein operate at the application layer of the Open System Interconnection (OSI) model and are represented by other OSI model layers. Built to rely on other software and / or hardware to provide network services.

  In addition, embodiments may deduplicate backed up data to more efficiently utilize available computing resources. According to some embodiments, data deduplication may be performed inline, that is, while the data storage system is receiving data to be deduplicated and stored. In other embodiments, data deduplication may be performed offline, i.e. after the data storage system has already stored the data to be deduplicated. As described in further detail below, embodiments may intelligently guide a variety of conventional and unconventional deduplication techniques to provide highly scalable deduplication services.

  A storage system according to aspects of the invention includes hardware and software that interface with a host computer (running a backup / restore application) and a backup storage medium. The storage system emulates tape or other types of removable storage media so that the backup / restore application sees the same display of devices and media as a physical tape library, and linear and sequential tape format data. It can be designed to convert data suitable for storage on a random access disk. Thus, the storage system of the present invention does not require new backup / restore application software or policies (such as allowing a user to search for individual backup user files, as described below). May provide improved functionality.

  With reference to FIG. 2, an example of a networked computing environment including a backup storage system 170 in accordance with aspects of the invention is shown in block diagram form. As shown, host computer 120 is coupled to storage system 170 via network connection 121. The network connection 121 may be, for example, a fiber channel connection that enables high-speed transfer of data between the host computer 120 and the storage system 170. The host computer 120 may be or include one or more application servers 102 (see FIG. 1) and / or media servers 114 (see FIG. 1), any existing in a networked computing environment. It should be appreciated that data may be backed up from other computers or from the primary storage device 106 (see FIG. 1). In addition, one or more user computers 136 may also be coupled to the storage system 170 via another network connection 138, such as an Ethernet connection. As described in detail below, the storage system may allow a user of the user computer 136 to view and optionally restore a backup user file from the storage system.

  The storage system includes a backup storage medium 126, which may be, for example, one or more disk arrays, as described in more detail below. The backup storage medium 126 provides a real storage space for backup data from the host computer 120. However, the storage system 170 also removes removable media such as a tape library so that the backup / restore application running on the host computer 120 will appear as if the data is being backed up to a conventional removable storage medium. Software and additional hardware that emulates the storage system may be included. Thus, as shown in FIG. 2, the storage system 170 may include an “emulated medium” 134 that represents a removable storage medium such as, for example, a virtual or emulated tape. These “emulated media” 134 are presented to the host computer by storage system software and / or hardware and appear to the host computer 120 as physical storage media. As will be described in detail below, it is further interfaced between the emulated medium 134 and the actual backup storage medium 126 that is data from the storage system controller (not shown) and the host computer 120. And the switching network 132 that stores the data on the backup storage medium 126. Thus, the storage system “emulates” a conventional tape storage system for the host computer 120.

  According to one embodiment, the storage system may include a “logical metadata cache” 242 that stores metadata about user data backed up on the storage system 170 from the host computer 120. As used herein, the term “metadata” refers to data that represents information about user data and describes the attributes of real user data. A non-limiting exemplary list of metadata about a data object includes the data object size, the logical and / or physical location of the data object in primary storage, the creation date of the data object, the last modification date of the data object, the data object It may include a stored backup policy name, eg an identifier such as the name or watermark of the data object, and a data type of the data object such as a software application associated with the data object. The logical metadata cache 242 allows the user and / or software application to randomly place backup user files, compare user files with each other, or otherwise access and manipulate backup user files. Represents a searchable collection. Two examples of software applications that may use data stored in the logical metadata cache 242 include a synthetic full backup application 240 and an end-user restore application 300, more fully described below. In addition, the deduplication inductor described in more detail below may use metadata to provide a deduplication service that is extensible within the storage system.

  In overview, the synthetic full backup application 240 can create a synthetic full backup data set from one existing full backup data set and one or more incremental backup data sets. Synthetic full backups can eliminate the need for periodic (eg weekly) full backups, thereby saving considerable time and network resources. Details of the synthetic full backup application 240 are further described below. Similarly, the end user restore application 300, described in further detail below, allows an end user (eg, an operator of the user computer 136) to browse, locate, and browse previously backed up user files from the storage system 170. And / or to be able to restore.

  As described above, the storage system 170 includes hardware and software that interfaces with the host computer 120 and the backup storage medium 126. Both the hardware and software of the embodiments of the present invention appear to be backed up on tape from the perspective of the host computer 120, but are actually on separate storage media, eg, multiple disk arrays. A conventional tape library backup system may be emulated so that data is backed up.

  With reference to FIG. 3, an example of a storage system 170 in accordance with an aspect of the invention is shown in block diagram form. In one example, the storage system 170 hardware includes a storage system controller 122 and a switching network 132 that connects the storage system controller 122 to a backup storage medium 126. The storage system controller 122 includes a processor 127 (which may be a single processor or multiple processors) and a memory 129 (eg, RAM, ROM, PROM, etc.) that may execute all or part of the storage system software. EEPROM, flash memory, etc., or a combination thereof). Memory 129 may also be used to store metadata regarding data stored on backup storage medium 126. Software including programming code that implements embodiments of the invention is typically stored on a computer readable and / or writable non-volatile recording medium, such as RAM, ROM, optical disk or magnetic disk, or tape. Is then copied to memory 129 where it is then executed by processor 127. Such programming code may be, for example, Assembler, Java (registered trademark), Visual Basic, C, C #, or C ++, Fortran, Pascal, Eifel ( It may be written in any of several programming languages such as Eiffel), Basic, Basic, COBOL, or a combination thereof. This is because the present invention is not limited to a specific programming language. Typically, in operation, the processor 127 is a RAM, etc., in which data such as code implementing an embodiment of the invention is accessed from a non-volatile recording medium more quickly by the processor than the non-volatile recording medium. To be read into another form of memory.

  As shown in FIG. 3, the controller 122 also includes a number of port adapters that connect the controller 122 to the host computer 120 and to the switching network 132. As shown, host computer 120 is coupled to the storage system via port adapter 124a, which may be, for example, a fiber channel port adapter. Through the storage system controller 122, the host computer 120 can back up data on the backup storage medium 126 and recover data from the backup storage medium 126.

  In the illustrated example, the switching network 132 may include one or more Fiber Channel switches 128a, 128b. The storage system controller 122 includes a plurality of fiber channel port adapters 124b and 124c that couple the storage system controller to the fiber channel switches 128a, 128b. The storage system controller 122 allows the data to be backed up on the backup storage medium 126 via the fiber channel switches 128a, 128b. As shown in FIG. 3, the switching network 132 further includes one or more Ethernet switches 130a, 130b coupled to the storage system controller 122 via Ethernet port adapters 125a, 125b. May be. In one example, the storage system controller 122 includes another Ethernet port adapter 125C that can be coupled to, for example, the LAN 103 so that the storage system 170 can communicate with a host computer (eg, a user computer), as described below. In addition.

  In the example shown in FIG. 3, the storage system controller 122 is coupled to the backup storage medium 126 via a switching network that includes two Fiber Channel switches and two Ethernet switches. Providing at least two different switches in the storage system 170 eliminates a single point of failure in the system. In other words, even if one switch (eg, Fiber Channel switch 128a) fails, the storage system controller 122 can still communicate with the backup storage medium 126 via another switch. Such a configuration may be advantageous in terms of reliability and speed. For example, as described above, reliability is improved by installing redundant components and eliminating single points of failure. In addition, in some embodiments, the storage system controller can back up data on the backup storage medium 126 using some or all of the Fiber Channel switches in parallel, thereby providing overall Increase backup speed. However, it should be recognized that it is not required that the system include more than one of each type of switch, and that the switching network include both Fiber Channel switches and Ethernet switches. . Further, in the example where the backup storage medium 126 comprises a single disk array, no switch is required.

  As described above, in one embodiment, the backup storage medium 126 may include one or more disk arrays. In a preferred embodiment, the backup storage medium 126 includes a plurality of ATA or SATA disks. Such disks are “off-the-shelf” products and may be relatively inexpensive compared to conventional storage array products from manufacturers such as EMC, IBM. In addition, given the cost of removable media (eg tape) and the fact that such media has a limited service life, such disks are comparable in cost to traditional tape-based backup storage systems. It is. In addition, such discs can read / write data substantially faster than tape. For example, on a single Fiber Channel connection, data can be backed up on disk at a rate of at least about 150 MB / s, which translates to about 540 GB / hr, which is significantly higher than the tape backup rate (eg, 1 Digits) fast. In addition, several Fiber Channel connections may be realized in parallel, thereby further increasing the speed. According to one embodiment of the present invention, the backup storage medium may be organized to implement any of a number of RAID (Redundant Array of Independent Disks) schemes. For example, in one embodiment, the backup storage medium may implement a RAID-5 implementation.

  As described above, embodiments of the present invention emulate a conventional tape library backup system using a disk array to replace a tape cartridge as a physical backup storage medium, thereby providing a “virtual tape library” To do. The physical tape cartridge that was in the conventional tape library is replaced by what is referred to herein as a “virtual cartridge”. For the purposes of this disclosure, it is recognized that the term “virtual tape library” refers to an emulated tape library that may be implemented in software and / or physical hardware, for example, as one or more disk arrays. Should be. Furthermore, although this description primarily refers to emulated tapes, the storage system can also emulate other storage media such as CD-ROM or DVD-ROM, and the term “virtual cartridge” is generally It should be appreciated that it refers to an emulated storage medium, such as an emulated tape or an emulated CD. In one embodiment, the virtual cartridge actually corresponds to one or more hard disks.

  Thus, in one embodiment, a backup / restore application is provided with a software interface to emulate a tape library so that the data appears to be being backed up on tape. However, the actual data library is replaced with one or more disk arrays so that the data is actually backed up on these disk arrays. It should be appreciated that other types of removable media storage systems may be emulated and the invention is not limited to emulation of tape library storage systems. The following description now describes various aspects, features, and operations of software included in the storage system 170.

  Although the software is described as “included” in the storage system 170 and may be executed by the processor 127 of the storage system controller 122 (see FIG. 3), it is understood that all software is executed on the storage system controller 122. It should be recognized that it is not required. Software programs such as synthetic full backup applications and end-user restore applications may be executed on the host computer and / or user computer, some of which span all or part of the storage system controller, host computer, and user computer. It may be distributed. Thus, it should be appreciated that the storage system controller is not required to be a contained physical entity such as a computer. Storage system 170 may communicate with software that resides on a host computer, such as media server 114 or application server 102. In addition, the storage system may contain several software applications that may run on the same or different host computers, or may reside on those host computers. Furthermore, it should be appreciated that the storage system 170 is not limited to individual pieces of equipment, but in some embodiments, the storage system 170 may be embodied as individual pieces of equipment. In one example, the storage system 170 may be provided as a self-contained unit that acts as a “plug and play” replacement for a conventional tape library backup system (ie, no need to modify existing backup procedures and policies). Such storage system units may also be used in networked computing environments that include conventional backup systems to provide redundancy or additional storage capacity. In another example, the storage system 116 may be implemented in a distributed computing environment, such as a clustered environment or a grid environment.

  As described above, according to one embodiment, the host computer 120 (eg, the application server 102 or the media server 114, see FIG. 1) may be connected to a network link (eg, Data may be backed up on the backup storage medium 126 via the fiber channel link) 121. Although the following description primarily refers to backing up data to an emulated medium, the principles also apply to restoring backup data from the emulated medium. As described above, the flow of data between the host computer 120 and the emulated media 134 can be controlled by a backup / restore application. From the point of view of a backup / restore application, it can appear as if the data is actually being backed up on the physical version of the emulated media.

  Referring to FIG. 4, storage system software 150 represents an emulated medium and provides one or more interfaces between backup / restore application 140 resident on host computer 120 and backup storage medium 126. May include a logical abstraction layer. Software 150 accepts tape format data from backup / restore application 140 and converts the data into data suitable for storage on a random access disk (eg, hard disk, optical disk, etc.). In one example, the software 150 may be executed on the processor 127 of the storage system controller 122 and stored in the memory 129 (see FIG. 3).

  According to one embodiment, software 150 may provide SCSI emulation of a robotic mechanism used to transport tapes to and from tapes, tape drives, and tape drives, where a virtual tape library (virtual tape library) is provided. library: VTL) layer 142 may be included. The backup / restore application 140 may communicate with the VTL 142 using, for example, a SCSI command represented by arrow 144 (eg, backup or write data to the emulated media). Thus, the VTL may form a software interface between other storage system software and hardware and the backup / restore application, presenting the emulated storage medium 134 (FIG. 2) to the backup / restore application. However, the emulated media is made visible to the backup / restore application as a conventional removable backup storage medium.

  A second software layer, referred to herein as file system layer 146, may provide an interface between the emulated storage medium (represented as VTL) and physical backup storage medium 126. In one example, the file system 146 communicates with the backup storage medium 126 using, for example, SCSI commands represented by arrows 148 to read data from and write data to the backup storage medium 126. Acts as

  In one embodiment, VTL provides comprehensive tape library support and may support any SCSI media changer. Emulated tape devices may include, but are not limited to, IBM LTO-1 and LTO-2 tape devices, Quantum Super DLT320 tape devices, Quantum P3000 tape library systems, or STORAGETEK L180 tape library systems. . Within the VTL, each virtual cartridge is a file that can grow dynamically as data is stored. This is in contrast to a conventional tape cartridge with a fixed size. As described further below with respect to FIG. 5, one or more virtual cartridges may be stored in the system file.

  FIG. 5 shows an example of a data structure in the file system software 146 showing the system file 200 according to one embodiment of the present invention. In this embodiment, the system file 200 includes a header 202 and data 204. The header 202 may include information identifying each virtual cartridge stored in the system file. The header may also contain information such as whether the virtual cartridge is write-protected, the creation date / modification date of the virtual cartridge, and the like. In one example, the header 202 includes information that uniquely identifies each virtual cartridge and distinguishes each virtual cartridge from other virtual cartridges stored in the storage system. For example, this information may include the name and identification number of the virtual cartridge (corresponding to a barcode normally present on the physical tape so that the tape can be identified by a robotic mechanism). The header 202 may also include additional information such as the capacity of each virtual cartridge and the last modification date.

  According to one embodiment of the present invention, the size of the header 202 depends on the type of data stored (eg, a virtual cartridge representing a data backup from one or more host computer systems) and such that the system can track. May be optimized to reflect the number of distinct sets of data (eg, virtual cartridges). For example, data that is normally backed up to a tape storage system is typically characterized by a larger data set representing a large number of system files and user files. Since the data set is very large, the number of individual data files to be tracked is accordingly small. Thus, in one embodiment, the size of the header 202 is such that it stores too much data and cannot be tracked sufficiently (ie, the header is too large) and there is not enough space to store a sufficient number of cartridge identifiers (ie, The header may be too small). In one exemplary embodiment, header 202 utilizes the first 32 MB of system file 200. However, it will be appreciated that the header 202 may have different sizes based on system requirements and characteristics, and that different sizes may be selected for the header 202 depending on system requirements and capacity. Should.

  It should be appreciated that from a backup / restore application perspective, virtual cartridges appear as physical tape cartridges that all have the same attributes and characteristics. That is, to the backup and restore application, the virtual cartridge appears as a sequentially written tape. However, in a preferred embodiment, the data stored in the virtual cartridge is not stored in sequential format on the backup storage medium 126. Rather, the data that appears to be written to the virtual cartridge is actually stored in a file in the storage system as randomly accessible disk format data. Metadata is used to link the stored data to the virtual cartridge so that the backup / restore application can read and write data in the cartridge format.

  Thus, in an overview of a preferred embodiment, user data and / or system data (referred to herein as “file data”) is received by the storage system 170 from the host computer 120 and constitutes a backup storage medium 126. Remembered above. Storage system software 150 (see FIG. 4) and / or hardware writes this file data to backup storage medium 126 in the form of a system file, as will be described in more detail below. Metadata is extracted as the data files are backed up by the storage system controller to track the attributes of the backed up user files and / or system files. For example, such metadata for each file may include the file name, the date the file was created or last modified, encryption information about the file, and other information. In addition, for each file, metadata that links the file to the virtual cartridge may be created by the storage system. Using such metadata, the software provides emulation of the tape cartridge to the host computer. However, as will be described below, the file data is not actually stored in tape format, but rather is stored in a system file. Storing data in system files rather than sequential cartridge format allows quick, efficient and random access to individual files without having to scan the sequential data to find a particular file. Can be advantageous.

  As described above, according to one embodiment, file data (ie, user data and / or system data) is stored as system files on a backup storage medium, each system file including a header and data, where the data is a real user. Files and / or system files. The header 202 of each system file 200 includes a tape directory 206 containing metadata that links the user file and / or system file to the virtual cartridge. The term “metadata” as used herein refers to data describing attributes of real user data and / or system data, not user file data or system file data. According to one example, the tape directory may define the layout of data on the virtual cartridge down to the byte level.

  In one embodiment, the tape directory 206 has a table structure as shown in FIG. The table includes a column 220 for the type of information stored (eg, data, file marker (FM), etc.), a column 222 for the size in bytes of the used disk blocks, and the number of disk blocks that store the file data. A counting column 224. Thus, the tape directory allows the controller to randomly access any data file stored on the backup storage medium 126 (as opposed to sequentially). For example, referring to FIG. 6, data file 226 can be quickly located on a virtual tape. This is because the tape directory indicates that the data of the file 226 starts with one block from the beginning of the system file 200. This one block has no size. This is because it corresponds to a file marker (FM). File markers are not stored in system files. That is, the file marker corresponds to zero data. Since file markers are used by traditional tapes, the tape directory contains file markers, so backup / restore applications can write file markers along with data files and see file markers when browsing virtual cartridges. Expect. Thus, file markers are tracked in the tape directory. However, the file marker does not represent any data and is therefore not stored in the data section of the system file. For this reason, the data in the file 226 starts at the beginning of the data division of the system file indicated by the arrow 205 (see FIG. 5) and has a length of 1024 bytes (that is, one disk block having a size of 1024 bytes). It should be appreciated that other file data may be stored in block sizes other than 1024 bytes, depending on the amount of data, ie the size of the data file. For example, a larger data file may be stored using a larger data block size for efficiency.

  In one example, the tape directory may be included in a “file descriptor” associated with each data file backed up on the storage system. The file descriptor includes metadata about the data file 204 stored on the storage system. In one embodiment, the file descriptor may be implemented according to a standardized format, such as the tape archive (tar) format used by many UNIX-based systems. Each file descriptor may include information such as the name of the corresponding user file, the date when the user file was created / modified, the size of the user file, and access restrictions on the user file. The additional information stored in the file descriptor may further include information describing a directory structure from which data is copied. Thus, as will be described in more detail below, the file descriptor may include searchable metadata about the corresponding data file.

  From the point of view of a backup / restore application, any virtual cartridge may include multiple data files and corresponding file descriptors. From the perspective of storage system software, data files are stored in system files that can be linked to a particular backup operation, for example. For example, a backup performed by one host computer at a particular time may generate one system file that may correspond to one or more virtual cartridges. The virtual cartridge can thus be of any size and can grow dynamically as more user files are stored on the virtual cartridge.

  Referring back to FIG. 2, as described above, the storage system 170 may include a synthetic full backup software application 240. In one embodiment, host computer 120 backs up data on emulated medium 134 to form one or more virtual cartridges. In some computing environments, a “full backup”, ie a backup copy of all data stored on the primary storage system of the network (see FIG. 1), may be performed periodically (eg weekly). This process is usually very long due to the large amount of data to be copied. Thus, in many computing environments, additional backups, called incremental backups, may be performed daily, for example, between successive full backups. An incremental backup is a process that backs up only the data that has changed since the last backup (whether incremental or full). Typically, this changed data is backed up on a file basis, even though most of the data in the file is often unchanged. For this reason, incremental backups are usually much smaller than full backups and are therefore performed much more quickly. It should be appreciated that many embodiments typically perform a full backup once a week and perform incremental backups daily for that week, but are not required to use such time frames. is there. Thus, some environments may require incremental backups several times a day. The principles of the invention apply to any environment that uses full backups (and optionally incremental backups) regardless of how often they are performed. Frequent full backups and / or incremental backups may result in a large amount of redundant data being stored in the storage system 170. To alleviate the burden associated with this redundant data, the storage system 170 may utilize a data deduplication system and process described further below.

  During the full backup procedure, the host computer may create one or more virtual cartridges containing backup data comprising a plurality of data files. For clarity, the following description assumes that a full backup generates only one virtual cartridge. However, it should be appreciated that a full backup can generate more than one virtual cartridge, and that the principles of the invention apply to any number of virtual cartridges.

  According to one embodiment, a method is provided for creating a synthetic full backup data set from one existing full backup data set and one or more incremental backup data sets. This method can eliminate the need for periodic (eg weekly) full backups, thereby saving considerable time and network resources for the user. Further, as is known to those skilled in the art, restoring data based on one full backup and one or more incremental backups can be a time consuming process. For example, if the latest version of a file is in an incremental backup, the backup / restore application typically restores the file based on the last full backup and then applies changes from the incremental backup. Thus, providing a synthetic full backup allows backup restore applications to restore data files more quickly based on only synthetic full backups without having to restore multiple times from one full backup and one or more incremental backups It can have the additional advantage of being able to. The expression “latest version” as used herein generally refers to the latest copy of a data file (ie, the last time the data file was saved), regardless of whether the file has a new version number. Point. The term “version” is generally used herein to refer to a copy of the same file that may have been modified in some way or stored multiple times.

  Referring to FIG. 7, a schematic diagram of a synthetic full backup procedure is shown. The host computer 120 may perform a full backup 230 at a first time, for example, at the weekend. The host computer 120 may then perform subsequent incremental backups 232a, 232b, 232c, 232d, 232e, for example daily for the week. The storage system 170 may then create a synthetic full backup data set 234 as described below.

  According to one embodiment, the storage system 170 may include a software application referred to herein as a synthetic full backup application 240 (see FIG. 3). The synthetic full backup application 240 may be executed on the storage system controller 122 (see FIG. 2) or may be executed on the host computer 120. The synthetic full backup application includes the software commands and interfaces necessary to create a synthetic full backup data set 234. In one example, the synthetic full backup application performs a logical merge of the metadata representations of each of the full backup data set 230 and the incremental backup data set 232 to create a new virtual cartridge that includes the synthetic full backup data set 234. Also good.

  For example, referring to FIG. 8, an existing full backup data set may include user files F1, F2, F3, and F4. The first incremental backup data set 232a may include a user file F2 ′ that is a modified version of F2, and F3 ′ that is a modified version of F3. The second incremental backup data set 232b may include a user file F1 ′ that is a modified version of F1, a further modified version of F2, F2 ″, and a new user file F5. A synthetic full backup data set 234 formed from a logical merge of the backup data set 230 and the two incremental data sets 232a and 232b includes a final version of each user file F1, F2, F3, F4, and F5. As shown in FIG. 8, the composite full backup data set thus includes user files F1 ′, F2 ″, F3 ′, F4, and F5.

  Referring back to FIGS. 3 and 4, the file system software 146 may create a logical metadata cache 242 that stores metadata about each user file stored on the emulated medium 134. It should be appreciated that the logical metadata cache need not be a physical data cache, but instead can be a searchable collection of data stored on the storage medium 126. In another example, the logical metadata cache 242 can be implemented as a database. When metadata is stored in the database, the logical combination of the full backup set and one or more incremental backup data sets to create a synthetic full backup data set using conventional database commands (eg, SQL commands) Mergers can be performed.

  In another embodiment, some of the metadata may be stored in a database and another portion may be stored in a storage system file. For example, backup data set metadata including the name of the backup data set and the data objects it comprises may be included in a traditional database, while the data object is a data file, data file size, security information, Also, metadata specific to the data object, such as in the case of a location in primary storage, may be included in the storage system file. Storing metadata in this way allows data that has been frequently queried to be retrieved flexibly from traditional databases, and data that has not been queried frequently is stored more quickly in storage system files. To promote system extensibility.

  As described above, each data file stored on emulated medium 134 may include a file descriptor that includes metadata about the data file, including the location of the file on backup storage medium 126. In one embodiment, a backup / restore application running on the host computer 120 stores the data on the emulated media 134 in a streaming tape format. An example of the data structure 250 representing this tape format is shown in FIG. As mentioned above, the system file data structure includes a header, which includes a file descriptor for the data file, the creation and / or modification date of the file, security information, the directory structure of the host system from which the file was obtained, And information about the data file, such as other information linking the file to the virtual cartridge. These headers are associated with data 254 which is a real user file and a system file backed up (copied) from a host computer, a primary storage system, or the like. The system file data structure may also optionally include a pad 256 that can properly align the next header to the block boundary.

  As shown in FIG. 9, in one example, the header data is located in the logical metadata cache 242 to allow for quick retrieval and random access to an otherwise sequential tape data format. The use of a logical metadata cache, implemented using file system software 146 for storage system controller 122, allows linear and sequential tape data formats stored on emulated media 134 to be backed up as storage media. 126 can be converted to a random access data format stored on the physical disks that make up 126. The logical metadata cache 242 includes a header 252 that includes a file descriptor for the data file, security information that can be used to control access to the data file, as described in more detail below, virtual cartridges and backups. A pointer 257 to the actual position of the data file on the storage medium 126 is stored. In one embodiment, the logical metadata cache stores data for all data files backed up in the full backup data set 230 and each incremental data set 232.

  According to one embodiment, the synthetic full backup application software 240 uses the information stored in the logical metadata cache to create a synthetic full backup data set. This composite full backup data set is then linked to the composite virtual cartridge created by the composite full backup application 240. To the backup / restore application, it appears as if the synthetic full backup data set is stored on this synthetic virtual cartridge. As described above, a synthetic full backup data set can be created by performing a logical merge of an existing full backup data set and an incremental backup data set. This logical merging involves comparing each data file contained in each of the existing full backup data set and incremental backup data set and, as described above with reference to FIG. 8, a composite of the latest version of each user file. It may include creating a body.

  According to one embodiment, the synthetic virtual cartridge 260 is a location of a data file on another virtual cartridge, specifically a virtual cartridge containing existing full and incremental backup data sets, as shown in FIG. Contains a pointer to. Considering the example listed above with respect to FIG. 8, the synthetic virtual cartridge 260 is an existing full backup on the virtual cartridge 262 of the user file F4 (since the existing full backup data set included the final version of F4). It includes a pointer 266 that points to the position in the data set and the position in the incremental data set 232a on the virtual cartridge 264, for example the user file F3 '(as indicated by arrow 268).

  The composite virtual cartridge also includes a list 270 that includes the identification numbers (and optionally names) of all virtual cartridges that contain the data pointed to by pointer 266. This subordinate cartridge list 270 can be important for references such as keeping track of the location of real data and to prevent subordinate virtual cartridges from being erased. In this example, the composite full backup data set does not contain real user files, but rather includes a set of pointers that indicate the location of the user files on the backup storage medium 126. Thus, it may be desirable to prevent real user files (stored on other virtual cartridges) from being deleted. This can be accomplished in part by keeping a record of the virtual cartridges containing data (subordinate cartridge list 270) and protecting each of those virtual cartridges from being overwritten or deleted. The composite virtual cartridge may also include cartridge data 272 such as the size of the composite virtual cartridge and its location on the backup storage medium 126. In addition, the composite virtual cartridge may have an identification number and / or name 274.

  According to another embodiment, the composite virtual cartridge may include a combination of a pointer and a stored real user file. Referring to FIG. 11, in one example, the composite virtual cartridge is a pointer that points to the location of the data file (the final version as described above with reference to FIG. 9) in the existing full backup data set 230 on the virtual cartridge 262. 266. The composite virtual cartridge may also include data 278 that includes actual data files copied from incremental data set 232 as indicated by arrow 280. In this way, it is possible to delete the incremental backup data set after creation of the synthetic full backup data set 276, thereby saving storage space. Synthetic virtual cartridges are relatively small because they contain all or part of a pointer rather than a copy of all user files.

  It should be appreciated that a synthetic full backup may include any combination of pointers and stored file data and is not limited to the examples listed above. For example, a synthetic full backup may include pointers to data files for some incremental backups and / or several files stored on the full backup, and other existing full backups and / or incremental backups It may also contain stored file data copied from. In addition, synthetic full based on previous full backups and associated incremental backups that do not include pointers, but include the final version of the actual file data copied from the appropriate full and / or incremental backups. A backup may be created.

  In one embodiment, the synthetic full backup application software uses user and system file metadata for each of the existing full and incremental backup data sets to determine where the final version of each data file is located. It is possible to include a difference algorithm that makes it possible to compare. For example, the diff algorithm compares the creation date and / or modification date, version number (if available), etc. between different versions of the same data file in different backup sets, and selects the latest version of the data file It can be used. However, users often open a user file and save the file without actually changing the data in the file (thus changing its modification date). Thus, the system may implement a more advanced difference algorithm that can analyze data in system files or user files to determine whether the data has actually changed. Variations of such difference algorithms, and other types of comparison algorithms, may be known to those skilled in the art. In addition, as described above, when metadata is stored in a database format, database commands such as SQL commands can also be used to perform logical merging. The present invention does so to ensure that the latest or last version of each user file is selected from all existing backup sets compared to properly create a synthetic full backup data set. Any of the various algorithms may be applied.

  As those skilled in the art will appreciate, synthetic full backup applications ensure that a full backup dataset is created and made available without requiring the host computer to perform a physical full backup. enable. This not only avoids imposing processor overhead on the host computer to transfer data to the backup storage system, but in embodiments where a synthetic full backup application is run on the storage system, it significantly reduces network bandwidth utilization. Decrease. As shown in FIG. 7, another synthetic full backup data set may be created using the first synthetic full backup data set 234 and the subsequent incremental backup data set 236. This can provide a significant time advantage in that files or objects that are not frequently modified are not frequently copied. Alternatively, the synthetic full backup data set may hold a pointer to a file that was copied only once.

  Some aspects according to the invention are directed to an expandable deduplication system that removes redundant data from data objects. For example, according to some embodiments, the deduplication system is configured to manage data deduplication using pre-processed metadata included in the data. More specifically, embodiments may direct data to the deduplication domain based on the presence or absence of specific metadata values in the data to be deduplicated. Each of these deduplication domains may employ specific deduplication techniques that are tailored to efficiently deduplicate specific types of data.

  For example, FIG. 14 depicts a block diagram of a deduplication inductor 1400 that is specifically configured to provide scalable deduplication services. The deduplication inductor 1400 may be implemented as software, hardware, or a combination thereof on various computer systems. For example, according to one embodiment, deduplication inductor 1400 is implemented as part of storage system controller 122 described above with respect to FIG. The particular configuration of the deduplication inductor 1400 shown in FIG. 14 is used for illustration only and is not intended to be limiting. This is because the embodiments of the present invention can be designed in various configurations without departing from the scope of the present invention. While some of the examples described herein focus on embodiments having a single deduplication inductor 1400, other embodiments may include two or more without departing from the scope of the present invention. A deduplication inductor may be included.

  Referring to FIG. 14, deduplication inductor 1400 includes a data interface 1402, a guidance engine 1404, a deduplication domain database 1406, deduplication domains 1408, 1410 and 1412, and a deduplication database interface 1414. In the illustrated example, data interface 1402 includes functions such as executable code, data, data structures or objects configured to exchange, eg, supply and receive information with one or more data sources. Also, in the illustrated example, the data interface 1402 can communicate with the guidance engine 1404 in both directions.

  As shown, guidance engine 1404 can exchange various information with data interface 1402, deduplication domain database 1406, and deduplication domains 1408, 1410 and 1412. Deduplication domain database 1406 may then communicate data with both guidance engine 1404 and deduplication database interface 1414. Deduplication database interface 1414 has functionality configured to exchange information with various external entities. These external entities may include users and / or systems, among others. In the illustrated example, deduplication database interface 1414 can also exchange information with deduplication domain database 1406. Each of the deduplication domains 1408, 1410 and 1412 includes functionality configured to exchange information with both the guidance engine 1404 and various external entities. For example, in one embodiment, deduplication domains 1408, 1410, and 1412 can exchange information with a data storage medium, such as backup storage medium 126 described with respect to FIG.

  Information may flow between the elements, components, and subsystems described herein using any technique. Such techniques include, for example, passing information over a network using a standard protocol such as TCP / IP, passing information between modules of memory, and files, databases, or some other non-volatile storage device Including passing information by writing to. In addition, instead of or in addition to a copy of the information, pointers or other references to the information may be sent and received. Conversely, information may be exchanged instead of or in addition to pointers or other references to information. Other techniques and protocols for communicating information may be used without departing from the scope of the invention.

  In the example shown in FIG. 14, deduplication domain database 1406 includes functionality configured to store and retrieve information describing attributes of one or more deduplication domains. An example of this information is for each deduplication domain, the amount of computing resources that belong to or are allocated to that deduplication domain, the specific deduplication method used by that deduplication domain, and the associated deduplication domain One or more specified data object characteristics. Deduplication domain database 1406 may also hold artifacts associated with the deduplication method used by the deduplication domain, eg, a hash table.

  The deduplication domain database 1406 may take the form of any logical configuration capable of storing information on a computer readable medium, including flat files, index files, hierarchical databases, relational databases, or object oriented databases. Good. Data may be modeled using unique and foreign key relationships and indexes. Unique and foreign key relationships and indexes may be established between the various fields and tables to ensure both data integrity and data exchange performance.

  In the illustrated example, the data interface 1402 includes functionality configured to exchange various types and formats of information with various data sources. These data sources may include sources of information subject to deduplication processing, such as primary storage device 106 described above with respect to FIG. The data interface 1402 can receive, among other data formats, discrete blocks of data, continuous data streams, and multiplexed data streams from multiple storage locations. In addition, the data interface 1402 can receive data inline, ie, data to be deduplicated and stored while the data storage system including the data interface 1402 is receiving, or offline, Data to be deduplicated can be received after the data storage device has already stored it.

  In the illustrated example, deduplication domains 1408, 1410, and 1412 each include one or more individual deduplication domains. The deduplication domain may include software and / or hardware having functionality configured to perform deduplication processing on data objects. Each deduplication domain may include dedicated data storage. In the illustrated example, each deduplication domain may be associated with one or more characteristics that are common to several data objects. In addition, in this example, each deduplication domain can employ a specific deduplication method. These features allow individual deduplication domains to provide a highly effective deduplication environment for the associated data objects.

  For example, according to one embodiment, deduplication domains 1408, 1410, and 1412 each employ a content recognition deduplication process, such as process 1200 described below. However, in other embodiments, deduplication domain 1408 may use hash fingerprint processing, deduplication domain 1410 may use pattern recognition processing, and 1412 may employ process 1200. Thus, the embodiments are not limited to a specific deduplication method or configuration of the deduplication method.

  In various embodiments, the guidance engine 1404 includes functionality configured to direct a data object to a deduplication domain associated with one or more characteristics of the data object. According to one embodiment, these characteristics include metadata associated with the data object. In the illustrated embodiment, the guidance engine 1404 can receive data objects from the data interface 1402. The guidance engine 1404 can select which of the deduplication domains 1408, 1410 and 1412 is suitable for deduplicating the received data object. As shown, guidance engine 1404 can also direct data objects to selected deduplication domains. Guidance engine 1404 also evaluates the results of the deduplication activities performed by deduplication domains 1408, 1410, and 1412, and evaluates the guidance engine based on this assessment to simplify redundant data across several deduplication domains. It has functionality configured to integrate into a single deduplication domain, thereby saving additional computing resources.

  In various embodiments, the guidance engine 1404 includes functionality configured to receive data from the data interface 1402 in a variety of formats and formats including discrete data blocks, data streams, and multiplexed data streams. In these embodiments, the guidance engine 1404 can extract pre-processed metadata from the received data. This metadata may include information of the type described above with respect to the logical metadata cache, so that in some embodiments, among other metadata, the backup policy name, data source type, data source It may include time series information such as name, backup application name, operating system type, data type, backup type, file name, directory structure, and date and time.

  Further, in some embodiments, the guidance engine 1404 has functionality configured to identify alignment points in the data stream or multiplexed data stream based on the extracted metadata. In these examples, the guidance engine 1404 can partition the data stream or multiplexed data stream along these alignment points to create a data object. Also, in some embodiments, the guidance engine 1404 can associate metadata with a data object. This associated metadata may include, among other things, metadata used to create the data object.

  For example, according to one embodiment, the guidance engine 1404 can align the data stream with the data object based on the data object being a subsequent backup of a particular server. Similarly, in another embodiment, the guidance engine 1404 can align data objects that include files with the same file name and directory location. In yet another embodiment, the guidance engine 1404 creates a data object, based on a policy executed by a backup / restore program to create the data object, or created by, for example, an Oracle database Metadata can be associated based on what type of data, such as data, is contained in the data object.

  According to one embodiment, the navigation engine 1404 has functionality configured to navigate the data object by evaluating metadata associated with the data object. In this example, the guidance engine 1404 can compare the metadata associated with the data object with the data object characteristics associated with the individual deduplication domain. If a sufficient quality match is found, the guidance engine 1404 can transfer the data object to the matching deduplication domain for further processing. According to one embodiment, the guidance engine 1404 can transfer a data object by providing a copy of the data object to the deduplication domain or by providing a reference to the data object, such as a pointer, to the deduplication domain. . According to some embodiments, the metadata associated with the data object and the data object characteristic associated with the deduplication domain are both information about the content of the data object.

  For example, in one embodiment, the metadata associated with the data object and the data object characteristics associated with the deduplication domain are software applications that created the data object, eg, Microsoft Outlook. In this example, upon encountering data objects created by Microsoft Outlook, the derivation engine 1404 can direct those data objects to the deduplication domain associated with the Microsoft Outlook data object. In other embodiments, the metadata and data object characteristics may be other types of information.

  According to some embodiments, the guidance engine 1404 includes functionality configured to further integrate redundant data across multiple deduplication domains. In some examples, the guidance engine can evaluate the results of the deduplication process and the artifacts associated with the deduplication process to determine redundant data across multiple deduplication domains. For example, in one embodiment, the guidance engine 1404 searches for hash fingerprints that may be shared by multiple deduplication domains and hashes tables associated with multiple deduplication domains that employ hash fingerprinting. Can be "scrubbed" or retrieved periodically. In this example, the guidance engine 1404 handles storage of data processed by different deduplication domains but having these common fingerprints, a copy of redundant data as a reference to a single copy of redundant data. Integration can be achieved by inducing one or more of the deduplication domains to replace.

  In other embodiments, the guidance engine 1404 creates a new deduplication domain or modifies the configuration of an existing deduplication domain to consolidate future processing of data regarding redundant data found by the scrubbing process described above. be able to. For example, in one embodiment, the guidance engine 1404 may further process data objects that contain data about redundant data from one deduplication domain by changing data object characteristics associated with a particular deduplication domain. , Can shift to another.

  For example, in one embodiment, the guidance engine 1404 includes functionality configured to find metadata common to data objects including redundant data found through scrubbing. Further, in this example, the guidance engine can determine one or more data object characteristics that match the common metadata based on the common metadata. Guidance engine 1404 also stores these associations in deduplication domain database 1406 to convert these newly determined data object characteristics into new or existing deduplication domains, ie deduplication domains into which future processing is integrated. Can be associated with Conversely, the guidance engine 1404 interacts with the deduplication domain database 1406 to separate one or more data object characteristics from existing deduplication domains, which will contain data about redundant data in the future. You can prevent data objects from being received. In this manner, the guidance engine 1404 can regulate the flow of data objects associated with newly found common metadata to the deduplication domain associated with the newly determined data object characteristics.

  In some embodiments, the guidance engine 1404 includes functionality configured to use additional information in directing data objects to a particular deduplication domain. For example, according to one embodiment, the guidance engine 1404 can detect that storage dedicated to a particular deduplication domain is below the remaining capacity threshold level. In this case, the guidance engine 1404 can direct the data object to another deduplication domain, or can allocate additional storage capacity to the deduplication domain. In another example, the guidance engine 1404 can direct the data object to a particular deduplication domain based on the amount of time remaining until the data object expires. For example, in this example, the guidance engine 1404 can direct a data object that has a low remaining lifetime to a deduplication domain with little processing overhead, regardless of the effectiveness of that deduplication domain for that data object. . This is because the data object is deleted from the storage in a short time.

  According to various embodiments, the deduplication database interface 1414 has functionality configured to exchange information with various external entities. In accordance with the illustrated embodiment, the deduplication database interface 1414 provides the user with various user interface metaphors that allow the user to create, modify, and delete deduplication domains, such as deduplication domains 1408, 1410, and 1412. Can be provided. More specifically, when displaying a metaphor for creating a new deduplication domain, the deduplication database interface 1414 is an interface that allows the user to identify characteristics of the data objects associated with the new deduplication domain. Elements can be presented to the user. In addition, the deduplication database interface 1414 can provide the user with interface elements that allow the user to specify the deduplication method to be employed by the new deduplication domain.

  In another embodiment, the deduplication database interface 1414 receives data from an external system such as a backup / restore program and automatically configures the deduplication domain to process data coming from the external system based on the received data. Having a function configured to configure. For example, in some embodiments, the deduplication database interface 1414 determines the commonality in the types of data objects that are received in the future or that are currently being received to increase the deduplication efficiency 1408, 1410 and 1412 can be configured. In one embodiment, the deduplication database interface 1414 can determine the primary storage location of data objects received as a result of executing a backup policy based on a backup policy provided by a backup / restore program. In this example, the deduplication database interface 1414 can store the configuration of the deduplication domains 1408, 1410 and 1412 based on this primary storage location information. In another example, the configuration stored by the deduplication database interface 1414 can be based on the software application that created the data object, rather than the storage location of the data object. Other embodiments may use other types of data to determine suitable deduplication domain structures and configurations.

  As described above, deduplication inductor 1400 may remove redundant data from a data object using one of several deduplication methods. One particular deduplication technique that can be used by a deduplication domain is content recognition deduplication. FIG. 12 illustrates an exemplary content recognition process 1200 for deduplicating data from a data object according to one embodiment of the present invention. FIG. 13 illustrates an evolved reference approach that, when used with data deduplication, creates additional processing efficiency. The deduplication process may be implemented using a single backup storage system or in a distributed storage system environment such as the grid environment described above.

  In general, the system performing process 1200 extracts those metadata objects that have undergone further deduplication processing steps, such as data objects that may extract metadata associated with a set of data objects, for example, data objects that may share duplicate data. May be identified. The system may examine data objects identified for additional processing to locate redundant data. In addition, the system may construct copies of the identified data objects that point to a single copy of the redundant data and optionally verify the integrity of these copies. In order to reclaim the storage capacity occupied by the redundant data, the system may delete the originally identified data object. Aspects and embodiments of the deduplication method and deduplication device are described in more detail below.

  With continued reference to FIG. 12, at step 1202, data deduplication processing 1200 begins. In step 1204, the system identifies data objects that are to undergo further deduplication processing. In one embodiment, the system may identify data objects that may contain redundant data. Various methods and metadata may be employed to perform this identification. For example, in one embodiment, the physical location of one backup data object in primary storage may indicate that it may have data with another backup data object. More specifically, if two backup data objects arise from the same primary storage device, eg, a particular server, these data objects may be identified as potentially containing redundant data copies. Similarly, in another embodiment, two data objects may be identified as potentially having redundant data if both were created by a particular software application. In yet another example, whether the data object was stored as part of a full or incremental backup policy may indicate the possibility of redundant data. Identification of data objects that may contain duplicate data is a process 1200 by allowing scarce computer resources, such as CPU cycles, to be concentrated on those data objects that will most benefit from the removal of redundant data. Increase the overall efficiency of

  In another embodiment, the system automatically includes or excludes certain data objects from further deduplication processing based on metadata associated with these data objects. It may be configured as follows. For example, the system may be configured to include data objects created by a particular software application in the deduplication process. Similarly, the system may be configured to include data objects backed up as part of a particular policy for further deduplication processing. Conversely, the system may be configured to exclude all data objects backed up by a particular policy and / or specifically named data objects from further deduplication processing. These configuration options allow system behavior to be tailored to suit the specific needs of any client environment, thus increasing system efficiency, performance, and scalability.

  At step 1206, the system performing process 1200 locates redundant data in the identified data object for further deduplication processing. This analysis may be accomplished by using metadata and / or by examining the actual content of the identified data object. In one embodiment, data objects having similar metadata are assumed to comprise the same data. For example, if multiple data objects are data files that both share the same name, primary storage and physical location in cyclic redundancy check (CRC), hashes generated during the deduplication process, or some other metadata, These two data objects may be recorded as redundant. Using metadata to identify redundant data provides several advantages. The use of metadata increases efficiency. This is because only the metadata of the data object can be processed rather than the entire data object.

  In another embodiment, the data objects may be compared bit by bit to ensure redundancy and then recorded as such. This type of comparison may use a lot of computing resources, but it also provides a strong guarantee that any data identified as redundant is actually fully redundant. This approach to redundancy determination may be useful when dealing with data objects where integrity is particularly important, for example financial information.

  In yet another embodiment, a portion of the data contained in the data object is analyzed to determine overall object redundancy. For example, some software applications may drive modified data to a location in the data object that they modify, for example at the beginning or end of the object. Thus, using this data distribution pattern, the system may concentrate its deduplication processing on the portion of the data object that is more likely to be static, thus increasing system efficiency.

  Embodiments of the present invention may employ a combination of these techniques to locate redundant data. More specifically, the system may direct a particular approach to a particular data object based on metadata such as that used to identify the data object for further deduplication processing as described above. . This metadata may include, among other things, the location in primary storage, the policy that caused the data object to be backed up, and the software application associated with the data object. Similar to data object identification, the ability to tune the system with respect to the manner in which duplicate data is located increases system scalability and performance.

  At step 1208, the system performing process 1200 may create a deduplicated copy of a previously identified data object that includes redundant data. These deduplicated copies may have little or no redundant data. In one embodiment, the identified data object may include, for example, a virtual cartridge. In this case, the system may create one or more deduplicated virtual cartridges that contain all the data contained in the virtual cartridges that are identified as being fully disassembled. Similar to the synthetic virtual cartridge described above, these deduplicated virtual cartridges may include both a data object and a pointer to the data object.

  During the creation of these deduplicated data copies, the system stores duplicate data copies in one particular data object, creates and / or modifies pointers in other data objects, May be stored in the data object. The system may follow various methodologies when storing duplicate data and pointers. In one embodiment, duplicate data is contained in the oldest data object, and a pointer identifying the location of the duplicate data is stored in the younger data object that contains the duplicate data. This technique, referred to in the art as backward referencing, is common when hash indexes are constructed to summarize data objects for deduplication processing.

  In another embodiment, duplicate data is contained in the youngest data object and a pointer identifying the location of the duplicate data is stored in an older data object that contains the duplicate data. This approach may be referred to as forward reference. Forward referencing enhances data restoration performance when data is restored from the last backup. This is because it is necessary to reduce the dereference of pointers in order to decompose all data included in the backup data object. This increase in performance is particularly advantageous due to the fact that the latest or youngest backup is usually used when data must be restored from primary storage.

  Figures 13A, 13B and 13C show forward and backward references as described above. FIG. 13A shows backup data objects 1302 and 1304 before deduplication processing. For purposes of this illustration, assume that backup data object 1302 was stored prior to backup data object 1304. Backup data object 1302 includes a unique data portion 1306 and a redundant data portion 1310A. Backup data object 1304 includes a unique data portion 1308 and a redundant data portion 1318B.

  FIG. 13B shows a deduplicated copy of data objects 1302 and 1304 under the forward reference scheme. Of the two, the more recently stored data object 1304 includes a copy of the redundant data portion 1310B. Of the two, the earlier stored data object 1302 includes a pointer 1312 that points to the redundant data portion 1310B. Thus, after making a deduplicated copy, the younger data object contains a copy of the redundant data and the older data object contains a pointer to the redundant data of the younger data object.

  FIG. 13C shows a deduplicated copy of data objects 1302 and 1304 under a reverse referencing scheme. Of the two, the earlier stored data object 1302 includes a copy of redundant data 1310A. Of the two, the more recently stored data object 1302 includes a pointer 1312 that points to the redundant data portion 1310A. Thus, after making a deduplicated copy, the older data object contains a copy of the redundant data and the younger data object contains a pointer to the redundant data of the older data object.

  At step 1210, the system may compare the deduplicated copy with a previously identified data object to ensure that data integrity has been preserved. This comparison may require dereferencing of the data object pointer and may include a bit-by-bit comparison of the data contained in the data object. After performing this integrity check, in one embodiment, the system exchanges pointers identifying the deduplicated copies with their respective previously identified data objects to ensure that the deduplicated data objects are It may be a primary data object so that a previously identified data object can be deleted without disrupting the integrity of the data object that references it. The system may also make other adjustments to the metadata to ensure that it accurately reflects the characteristics of the deduplicated copy.

  In step 1212, the storage capacity utilized by the previously identified data object is reclaimed for use by other data objects. In one embodiment, this may be accomplished by simply deleting a previously identified data object. At step 1214, process 1200 ends.

  Process 1200 illustrates a preferred sequence of events. Other actions can be added or the order of actions can be changed in process 1200 without departing from the spirit of the invention. In one embodiment, process 1200 may be performed for each data object included in the backup storage system. In another example, the system may perform process 1200 on a subset of data objects in the backup storage system.

  Process 1200 may be performed as desired, or may be scheduled as a one-time process or a repetitive process. If the space reclaimed by deduplication meets or exceeds a certain threshold, a further subset of process 1200 may be performed. For example, in one embodiment, process 1200 operates only if deduplication frees at least a certain number (eg, 50) terabytes, or a certain percentage (eg, 25%) of used backup storage capacity. May be. If implemented as an event-driven computing operation, the acts making up process 1200 may be performed in a distributed computing environment, such as a grid environment.

  Thus, in summary, the embodiment of the deduplication process 1200 reduces the storage capacity required to maintain a copy of the backup data, and thus reduces the amount of electronic media required to store the backup data. . Further, embodiments of the deduplication process 1200 may efficiently use computing resources by optimizing the deduplication process using metadata. Finally, by storing deduplicated data in a forward reference manner, deduplication can enhance the performance of commonly used data recovery functionality.

  Various embodiments include a process for a computer system that provides an extensible deduplication service. FIG. 15 illustrates an example of one such process 1500 that includes receiving data, selecting a deduplication domain to process the data, and directing the data to the selected deduplication domain. Process 1500 begins at 1502.

  At act 1504, the computer system receives data to be deduplicated. As described above, according to one embodiment, data may take various forms including blocks of data, data streams, and multiplexed data streams. In the example shown in FIG. 14, data is received by the data interface 1402 and provided to the guidance engine 1404 for further processing. According to this example, the guidance engine 1404 receives the data and partitions the data into one or more data objects based on pre-processed metadata included in the data stream. Further, in this example, the guidance engine 1404 associates metadata with the data object it creates.

  At act 1506, the computer system selects a deduplication domain to process the received data. According to the example shown in FIG. 14, the guidance engine 1404 processes a particular data object by comparing the metadata associated with that data object with the data object characteristics associated with the deduplication domain. Select one of the deduplication domains 1408, 1410 and 1412. In addition, the guidance engine 1404 may or may not select a particular deduplication domain based on other information, such as the amount of storage capacity remaining in the particular deduplication domain.

  At act 1508, the computer system directs the received data to the selected deduplication domain. According to the illustrated example of FIG. 14, the guidance engine 1404 provides a data object to the selected deduplication domain by passing a reference to the data object or a copy of the data object to the selected deduplication domain. May be.

The process 1500 ends at 1510.
Process 1500 illustrates a particular sequence of actions in a particular embodiment. The actions involved in each of these processes may be performed by or using one or more computer systems specially configured as described herein. In addition, the order of actions can be changed and other actions can be added without departing from the scope of the invention.

  As described above with reference to FIG. 3, the storage system may also include a software application called an end user restore application 300. For this reason, according to another embodiment, the end user locates the backup data and does not need to make any changes to existing backup / restore procedures and / or policies without IT staff intervention. A method for restoring is provided. In a typical backup storage system, the backup / restore application running on the host computer 120 is controlled by IT staff, and is it impossible for end users to access backup data without IT staff intervention? Or it can be very difficult. According to aspects and embodiments of the present invention, end users can locate their files and restore them, for example via a web-based or other interface having a backup storage medium 126 Storage system software is provided.

  As with the synthetic full backup application 240, it is recognized that the end user restore application 300 may be executed on the storage system controller 122 (see FIG. 2) or may be executed on the host computer 120. Should be. The end-user restore application is a software command required to allow an authenticated user to search the logical metadata cache to locate the backup file and optionally restore the backup file from the backup storage medium 126. And including the interface.

  According to one embodiment, software is provided that includes a user interface that is installed on and / or executed on the user computer 136. This user interface can be any type of interface that allows a user to locate a file on a backup storage medium. For example, the user interface may be a graphical user interface, web based, or a text interface. The user computer is coupled to the storage system 170 via a network connection 138, which can be, for example, an Ethernet connection. Through this network connection 138, the operator of the user computer 136 can access data stored on the storage system 170.

  In one example, the end user restore application 300 includes user authentication and / or certification features. For example, a user may be required to log in via a user interface on a user computer using a username and password. The user computer may communicate the username and password to a storage system (eg, an end user restore application) that may use an appropriate user authentication mechanism to determine whether the user has access to the storage system. Good. Some examples of user authentication mechanisms that can be used are Microsoft Active Directory server, Unix (Yellow) "Yellow Page" server, or Lightweight Directory Access Protocol. Including, but not limited to. The login / user authentication mechanism may communicate with the end-user restore application to exchange user rights. For example, some users may be allowed to search only files that they have created or that have certain rights or are identified as owners. Other users, such as system operators or administrators, may be allowed access to all backup files and the like.

  According to one embodiment, the end user restore application uses a logical metadata cache to obtain information about all data files backed up on a backup storage medium. The end-user restore application can be accessed via the user interface, eg by backup time / date, user name, original user computer directory structure (which may have been obtained when the file was backed up), or other file characteristics. The hierarchical directory structure of the classified user's files is presented to the user. In one example, the directory structure presented to a user can vary depending on the permissions enabled for that user. The end user restore application may accept browse requests (ie, through the user interface, the user may browse the directory structure to locate the desired file), or the user may be by name, date, etc. You may search for a file.

  According to one embodiment, the user may restore a backup file from the storage system. For example, once the user locates the desired file, the user may download the file from the storage system via the network connection 138 as described above. In one example, this download procedure may be implemented in a manner equivalent to a web-based download as is known to those skilled in the art.

  By enabling end users to access files that they are authorized to view / download, and by allowing such access through a user interface (eg, web-based technology), end-user restore applications can be backed up. Allow users to find and restore their own files without having to change policies or procedures.

  Although aspects of the present invention, such as synthetic full backup applications and end-user restore applications, are primarily described herein with respect to software, they can instead be implemented in software, hardware or firmware, or any combination thereof. It should be recognized that.

  Thus, for example, embodiments of the present invention provide a computer that at least partially performs the functions of a synthetic full backup application and / or end-user restore application as detailed above to a processor of a storage system at run time. A computer readable medium (eg, computer memory, floppy disk, compact disk, tape, etc.) encoded with a program (ie, a plurality of instructions) may be provided.

  In summary, the embodiments and aspects of the present invention thus emulate a conventional tape backup system, but can create synthetic backups and allow end users to browse and restore backup files. Including storage systems and methods that can provide improved functionality such as. However, it should be recognized that various aspects of the present invention may be used for purposes other than computer data backup. The storage system of the present invention may be used to economically store a huge amount of data, and the stored data can be accessed randomly rather than sequentially and with hard disk access time. This embodiment may find use outside of conventional backup storage systems. For example, embodiments of the present invention may be used to store video or audio data representing a wide selection of movies and music, allowing video and / or audio as needed.

  While several aspects of at least one embodiment of the present invention have been described, it should be recognized that various changes, modifications and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the scope of the invention. Accordingly, the foregoing description and drawings are merely exemplary.

Claims (29)

A method for inducing deduplication of application layer data objects, the method comprising:
Receiving the application layer data object;
Selecting a deduplication domain from a plurality of deduplication domains based at least in part on data object characteristics associated with the deduplication domain;
Determining that the application layer data object has the characteristics;
Directing the application layer data object to the selected deduplication domain. Receiving the application layer data object includes
Receiving a data stream;
The method of claim 1, comprising identifying the application layer data object using metadata included in the data stream.   The method of claim 2, wherein receiving the data stream comprises receiving a multiplexed data stream.   The method of claim 2, further comprising extracting metadata included in the data stream using the application layer data object.   Selecting the deduplication domain from the plurality of deduplication domains compares the extracted metadata associated with the application layer data object with at least one of the characteristics associated with the deduplication domain. The method of claim 4 comprising:   Extracting the metadata contained in the data stream includes backup policy name, data source type, data source name, backup application name, operation system type, data type, backup type, file name, directory structure, And extracting at least one of the time series information.   The method of claim 1, further comprising configuring each of the plurality of deduplication domains to use one of a plurality of deduplication methods.   Each of the plurality of deduplication domains is configured to use one deduplication method selected from the group comprising hash fingerprinting, pattern recognition, and content recognition deduplication. 8. The method of claim 7, comprising configuring each.   The method of claim 1, further comprising associating each of the plurality of deduplication domains with at least one data object characteristic. Deduplicating the application layer data object within the selected deduplication domain;
The method of claim 1, further comprising adjusting the data object characteristics associated with at least one of the plurality of deduplication domains based on a deduplication act result.   The method of claim 10, wherein adjusting the data object characteristics comprises storing data in a deduplication domain database.   A computer readable medium having stored thereon computer readable signals defining instructions that, when executed by a computer, instruct the computer to perform the method of claim 1.   The method of claim 1, wherein the method is executed in a grid computing environment.   The method of claim 1, wherein the method is performed on the backup storage system while data is not backed up to the backup storage system.   The method of claim 1, wherein the method is performed on the backup storage system while data is being backed up to the backup storage system. A system for inducing deduplication of application layer data objects, the system comprising:
A plurality of deduplication domains, each of the deduplication domains being associated with at least one characteristic common to a plurality of application layer data objects, the system further comprising:
A controller coupled to the plurality of deduplication domains, the controller comprising:
Receiving the application layer data object;
Determining that the application layer data object has the at least one characteristic associated with a deduplication domain;
A system configured to direct the application layer data object to the deduplication domain. The controller further includes:
Receive the data stream,
The system of claim 16, configured to identify the application layer data object using metadata contained in the data stream.   The system of claim 17, wherein the data stream is multiplexed.   The system of claim 17, wherein the controller is further configured to extract metadata contained in the data stream using the application layer data object.   The controller further indicates that the application layer data object has the at least one characteristic associated with the deduplication domain and that the extracted metadata associated with the application layer data object is the deduplication domain. The system of claim 19, wherein the system is configured to determine by comparing with the at least one characteristic associated with the.   The controller further includes at least one of a backup policy name, a data source type, a data source name, a backup application name, an operating system type, a data type, a backup type, a file name, a directory structure, and time series information. The system of claim 19, wherein the system is configured to extract.   The system of claim 16, wherein the controller is further configured to configure each of the plurality of deduplication domains to use one of a plurality of deduplication methods.   The controller is further configured to configure each of the plurality of deduplication domains to use one deduplication method selected from the group comprising hash fingerprinting, pattern recognition, and content recognition deduplication. The system of claim 22.   The system of claim 16, wherein the controller is further configured to associate each of the plurality of deduplication domains with at least one data object characteristic. The controller further includes:
Causing deduplication of the application layer data object within the selected deduplication domain;
The system of claim 16, wherein the system is configured to adjust the data object characteristic associated with at least one of the plurality of deduplication domains based on a deduplication act result.   26. The system of claim 25, wherein the controller is further configured to store data in a deduplication domain database.   The system of claim 16, included in a grid computing environment.   The controller further receives the application layer data object while data is not backed up to the system and determines that the application layer data object has the at least one characteristic associated with a deduplication domain. The system of claim 16, configured to direct the application layer data object to the deduplication domain.   The controller further receives the application layer data object while data is being backed up to the system and determines that the application layer data object has the at least one characteristic associated with a deduplication domain. The system of claim 16, configured to direct the application layer data object to the deduplication domain.

Images