US20100161554A1 - Asynchronous distributed de-duplication for replicated content addressable storage clusters - Google Patents

Asynchronous distributed de-duplication for replicated content addressable storage clusters Download PDF

Info

Publication number
US20100161554A1
US20100161554A1 US12/644,693 US64469309A US2010161554A1 US 20100161554 A1 US20100161554 A1 US 20100161554A1 US 64469309 A US64469309 A US 64469309A US 2010161554 A1 US2010161554 A1 US 2010161554A1
Authority
US
United States
Prior art keywords
index
replica
de
object
duplication
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US12/644,693
Other versions
US8712974B2 (en
Inventor
Gia DATUASHVILI
Alexander Kesselman
Alexandre Drobychev
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US13985708P priority Critical
Application filed by Google LLC filed Critical Google LLC
Priority to US12/644,693 priority patent/US8712974B2/en
Assigned to GOOGLE, INC. reassignment GOOGLE, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: DATUASHVILI, GIA, DROBYCHEV, ALEXANDRE, KESSELMAN, ALEXANDER
Publication of US20100161554A1 publication Critical patent/US20100161554A1/en
Publication of US8712974B2 publication Critical patent/US8712974B2/en
Application granted granted Critical
Assigned to GOOGLE LLC reassignment GOOGLE LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: GOOGLE INC.
Application status is Active legal-status Critical
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network-specific arrangements or communication protocols supporting networked applications
    • H04L67/10Network-specific arrangements or communication protocols supporting networked applications in which an application is distributed across nodes in the network
    • H04L67/1095Network-specific arrangements or communication protocols supporting networked applications in which an application is distributed across nodes in the network for supporting replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes or user terminals or syncML
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/174Redundancy elimination performed by the file system
    • G06F16/1748De-duplication implemented within the file system, e.g. based on file segments
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/178Techniques for file synchronisation in file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • G06F16/184Distributed file systems implemented as replicated file system
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2365Ensuring data consistency and integrity
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24553Query execution of query operations
    • G06F16/24554Unary operations; Data partitioning operations
    • G06F16/24556Aggregation; Duplicate elimination
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A method is performed by a device of a group of devices in a distributed data replication system. The method includes storing an index of objects in the distributed data replication system, the index being replicated while the objects are stored locally by the plurality of devices in the distributed data replication system. The method also includes conducting a scan of at least a portion of the index and identifying a redundant replica(s) of the at least one of the objects based on the scan of the index. The method further includes de-duplicating the redundant replica(s), and updating the index to reflect the status of the redundant replica.

Description

    RELATED APPLICATION
  • This application claims priority under 35 U.S.C. §119 based on U.S. Provisional Patent Application No. 61/139,857, filed Dec. 22, 2008, the disclosure of which is incorporated by reference herein in its entirety.
  • BACKGROUND
  • The enterprise computing landscape has undergone a fundamental shift in storage architectures in that central-service architecture has given way to distributed storage clusters. As businesses seek ways to increase storage efficiency, storage clusters built from commodity computers can deliver high performance, availability and scalability for new data-intensive applications at a fraction of the cost compared to monolithic disk arrays. To unlock the full potential of storage clusters, the data is replicated across multiple geographical locations, thereby increasing availability and reducing network distance from clients.
  • Data de-duplication can identify duplicate objects and reduce required storage space by removing duplicates. As a result, data de-duplication is becoming increasingly important for a storage industry and is being driven by the needs of large-scale systems that can contain many duplicates.
  • SUMMARY
  • According to one implementation, a method may be performed by a device of a group of devices in a distributed data replication system. The method may include storing an index of objects in the distributed data replication system, the index being replicated while the replicas of objects are stored locally by the plurality of devices in the distributed data replication system. The method may also include conducting a scan of at least a portion of the index and identifying a redundant replica of the at least one of the objects based on the scan of the index. The method may further include de-duplicating the redundant replica by writing a de-duplication record to a portion of the index.
  • According to another implementation, a device, of a group of devices in a distributed data replication system, may include means for storing an index of objects in the distributed data replication system; means for writing changes to the index to designate a status of a replica of one of the objects; means for replicating the changes to the index to the plurality of devices in the distributed data replication system; means for conducting a scan of at least a portion of the index; means for identifying a redundant replica of the one of the objects based on the scan of the index; and means for de-duplicating the redundant replica.
  • According to yet another implementation, a system may include a memory to store instructions, a data store of objects and an index of the objects in the data store; and a processor. The processor may execute instructions in the memory to identify a status of an object in the data store, the status relating to whether the object has a replica and whether a delete request is associated with the object, write a de-duplication designation record to the index based on the status of the object, replicate the index with the de-duplication designation record to one or more devices, and receive, from one of the one or more devices, other de-duplication designation records associated with the object, where the de-duplication designation record and the other de-duplication designation records provide a basis for deletion of one or more replicas of the object.
  • According to still another implementation, a method performed by one or more devices may include storing an index of objects in multiple devices within a distributed data replication system and replicating the index throughout the distributed data replication system while storing the objects locally, where each device is responsible for de-duplication of the objects within a particular subset of the index; conducting a scan of each of the subsets of the index to identify redundant replicas based on the scan; de-duplicating the redundant; and automatically copying an object from a device with a replica having an ongoing delete request to a device with a replica having been previously de-duplicated.
  • According to a further implementation, a computer-readable memory may include computer-executable instructions. The computer-readable memory may include one or more instructions to conduct a scan of a portion of a index of objects in a distributed data replication system; one or more instructions to identify a redundant replica of one of the objects based on the scan of the portion of the index; one or more instructions to de-duplicate the redundant replica.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments described herein and, together with the description, explain these embodiments. In the drawings:
  • FIG. 1 is a diagram of an exemplary network in which systems and methods described herein may be implemented;
  • FIG. 2 is a diagram of an exemplary configuration of the file system of FIG. 1;
  • FIG. 3 is a diagram of exemplary components of a storage cluster of FIG. 1;
  • FIG. 4 is a functional block diagram of an exemplary storage cluster of FIG. 1;
  • FIG. 5 is a diagram of an exemplary record structure that may be used within an index of a distributed multi-master data replication system;
  • FIGS. 6A-6B are flowcharts of exemplary processes for managing client-initiated upload/delete operations;
  • FIG. 7 is a flowchart of exemplary process for performing de-duplication in a distributed multi-master data replication system;
  • FIG. 8 is a flowchart of exemplary process for managing a delete request;
  • FIG. 9 is a flowchart of exemplary process for removing duplicate replicas;
  • FIG. 10 is a flowchart of exemplary process for optimizing bandwidth consumption and reducing latency in a distributed multi-master data replication system; and
  • FIG. 11 is a diagram illustrating a portion of an exemplary global index according to an implementation described herein.
  • DETAILED DESCRIPTION
  • The following detailed description refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Also, the following detailed description does not limit the invention.
  • Systems and/or methods described herein may provide an asynchronous distributed de-duplication algorithm for replicated storage clusters that provides availability, liveness and consistency guarantees for immutable objects. Implementations described herein may use the underlying replication layer of a distributed multi-master data replication system to replicate a content addressable index (also referred to herein as a “global index”) between different storage clusters. Each object of the global index may have a unique content handle (e.g., a hash value or digital signature). In implementations described herein, the removal process of redundant replicas may keep at least one replica alive.
  • Exemplary Network Configuration
  • FIG. 1 is a diagram of an exemplary system 100 in which systems and methods described herein may be implemented. System 100 may include clients 110-1 through 110-N (referred to collectively as clients 110, and individually as client 110) and storage clusters 120-1 through 120-M (referred to collectively as storage clusters 120, and individually as storage cluster 120) connected via a network 130. Storage clusters 120 may form a file system 140 (as shown by the dotted line in FIG. 1).
  • Network 130 may include one or more networks, such as a local area network (LAN), a wide area network (WAN), a telephone network (e.g., the Public Switched Telephone Network (PSTN)), an intranet, the Internet, a similar or dissimilar network, or a combination of networks. Clients 110 and storage clusters 120 may connect to network 130 via wired and/or wireless connections.
  • Clients 110 may include one or more types of devices, such as a personal computer, a wireless telephone, a personal digital assistant (PDA), a lap top, or another type of communication device, and/or a thread or process running on one of these devices. In one implementation, a client 110 includes, or is linked to, an application on whose behalf client 110 communicates with storage cluster 120 to read or modify (e.g., write) file data.
  • Storage cluster 120 may include one or more server devices, or other types of computation or communication devices, that may store, process, search, and/or provide information in a manner described herein. In one implementation, storage cluster 120 may include one or more servers (e.g., computer systems and/or applications) capable of maintaining a large-scale, random read/write-access data store for files. The data store of storage cluster 120 may permit an indexing system to quickly update portions of an index if a change occurs. The data store of storage cluster 120 may include one or more tables (e.g., a document table that may include one row per uniform resource locator (URL), auxiliary tables keyed by values other than URLs, etc.). In one example, storage cluster 120 may be included in a distributed storage system (e.g., a “Bigtable” as set forth in Chang et al., “Bigtable: A Distributed Storage System for Structured Data,” Proc. of the 7th OSDI, pp. 205-218 (November 2006)) for managing structured data (e.g., a random-access storage cluster of documents) that may be designed to scale to a very large size (e.g., petabytes of data across thousands of servers).
  • Although not shown in FIG. 1, system 100 may include a variety of other components, such as one or more dedicated consumer servers or hubs. A consumer server, for example, may store a read-only copy of a data store from one or more storage clusters 120 for access by clients 110. A hub, for example, may store a read-only copy of a data store from one or more storage clusters 120 for distribution to one or more consumer servers.
  • Exemplary Storage Cluster Configuration
  • FIG. 2 is a diagram of an exemplary configuration of the file system 140. As shown in FIG. 2, file system 140 may include storage clusters 120-1, 120-2, 120-3, and 120-4. In one implementation, file system 140 may be a distributed multi-master data replication system, where each of storage clusters 120-1, 120-2, 120-3, and 120-4 may act as a master server for the other storage clusters. In file system 140, data may be replicated across storage clusters 120-1, 120-2, 120-3, and 120-4 (e.g., in multiple geographical locations) to increase data availability and reduce network distance from clients (e.g., clients 110). Generally, distributed objects and references may be dynamically created, mutated, cloned and deleted in different storage clusters 120 and an underlying data replication layer (not shown) maintains the write-order fidelity to ensure that all storage clusters 120 will end up with the same version of data. Thus, the data replication layer respects the order of writes to the same replica for a single object.
  • A global index of all of the objects in the distributed multi-master data replication system may be associated with each storage cluster 120. Each stored object may be listed by a unique content handle (such as a hash value, digital signature, etc.) in the global index. Selected storage clusters may each be assigned to be responsible for a distinct range of the content handles in the global index. For example, a single storage cluster 120 may be responsible for de-duplication of objects associated with particular content handles. Changes to the global index made by one storage cluster may be replicated to other storage clusters.
  • Although FIG. 2 shows exemplary functional components of file system 140, in other implementations, file system 140 may contain fewer, additional, different, or differently arranged components than depicted in FIG. 2. In still other implementations, one or more components of file system 140 may perform one or more tasks described as being performed by one or more other components of file system 140.
  • FIG. 3 is a diagram of exemplary components of storage cluster 120. Storage cluster 120 may include a bus 310, a processor 320, a main memory 330, a read-only memory (ROM) 340, a storage device 350, an input device 360, an output device 370, and a communication interface 380. Bus 310 may include one or more conductors that permit communication among the components of storage cluster 120.
  • Processor 320 may include any type of processor or microprocessor that may interpret and execute instructions. Main memory 330 may include a random access memory (RAM) or another type of dynamic storage device that may store information and instructions for execution by processor 320. ROM 340 may include a ROM device or another type of static storage device that may store static information and instructions for use by processor 320. Storage device 350 may include a magnetic and/or optical recording medium and its corresponding drive. For example, storage device 350 may include one or more local disks 355 that provide persistent storage. In one implementation, storage cluster 120 may maintain metadata, for objects stored in file system 140, within one or more computer-readable mediums, such as main memory 330 and/or storage device 350. For example, storage cluster 120 may store a global index within storage device 350 for all the objects stored within a distributed multi-master data replication system.
  • Input device 360 may include one or more mechanisms that permit an operator to input information to storage cluster 120, such as a keyboard, a keypad, a button, a mouse, a pen, etc. Output device 370 may include one or more mechanisms that output information to the operator, including a display, a light emitting diode (LED), etc. Communication interface 380 may include any transceiver-like mechanism that enables storage cluster 120 to communicate with other devices and/or systems. For example, communication interface 380 may include mechanisms for communicating with other storage clusters 120 and/or clients 110.
  • FIG. 4 illustrates a functional block diagram of storage cluster 120. As shown in FIG. 4, storage cluster 120 may include data store 410 and de-duplication logic 420. In one implementation, as illustrated in FIG. 4, data store 410 may be provided within storage cluster 120. In other implementations, some or all of data store 410 may be stored within one or more other devices of system 100 in communication with storage cluster 120, such as external memory devices or devices associated with an indexing system (not shown).
  • Data store 410 may include a replicated index store 412 and a local object store 414. Replicated index store 412 may be included as part of the replication layer of the distributed multi-master data replication system. Replicated index store 412 may store information associated with the global index. At least a portion of replicated index store 412 may be replicated on multiple storage clusters 120. The number of replicas for each replicated index store 412 may be user-configurable. Local object store 414 may store objects locally within storage cluster 120. Local object store 414 may include files, such as images or videos uploaded by clients (e.g., clients 110).
  • De-duplication logic 420 may include logic to remove redundant replicas from storage clusters within the distributed multi-master data replication system (e.g., storage clusters 120-1, 120-2, 120-3, and 120-4). De-duplication logic 420 for each participating storage cluster may be assigned to be responsible for a particular section of the global index. For example, de-duplication logic 420 may be assigned to a particular range of content handles for the global index. Thus, only one storage cluster within the distributed multi-master data replication system may be able to perform destructive operations (e.g., deletion of replicas) on a replicated object within the system.
  • To facilitate de-duplication, records may be generated by de-duplication logic 420 and appended to a portion of the global index associated with a particular content handle. Records may include, for example, a “Data” designator for initiating a live replica, a “DeleteRequest” designator for indicating an ongoing delete request for a replica, and a “Deduped” designator for indicating a replica that has been selected for de-duplication. Record formats and uses are described in more detail below.
  • Although FIG. 4 shows exemplary functional components of storage cluster 120, in other implementations, storage cluster 120 may contain fewer, additional, different, or differently arranged functional components than depicted in FIG. 4. In still other implementations, one or more functional components of storage cluster 120 may perform one or more other tasks described as being performed by one or more other functional components.
  • Exemplary Record Structure
  • FIG. 5 provides an illustration of an exemplary record structure 500 for a de-duplication designation record that may be written to the global index in an exemplary implementation. The de-duplication designation record may be associated in the global index with a particular content handle of an object replica. As shown in FIG. 5, record structure 500 may include storage cluster identifier (“ID”) section 510, a storage location section 520, and designation section 530. Storage cluster identification section 510 may include a unique identification (e.g., “Cluster ID”) for the storage cluster 120 that is storing the object replica for which the record is being written. Location section 520 may include an address for the location of the replica within storage cluster 120 that is identified by storage cluster identification section 510. Designation section 530 may include, for example, a “Data” designator, a “DeleteRequest” designator, or a “Deduped” designator.
  • Record structure 500 may be listed in the form of “ClusterID:Location:Designation.” For example, a record for a replica may be added to the global index by storage cluster 120-1 with the record “01:234523/2000:DeleteRequest,” where “01” is the cluster ID for storage cluster 120-1, “234523/2000” is the location, within storage cluster 120-1 at which the replica is stored, and “DeleteRequest” is the designator. A record for another replica of the same object in storage cluster 120-2 may be “02:234544/1000:Data,” where “02” is the cluster ID for storage cluster 120-2, “234544/1000” is the location within storage cluster 120-2, and “Data” is the designator.
  • Exemplary Process Flows
  • FIGS. 6A and 6B are flowcharts of exemplary processes for managing client-initiated upload/delete operations. FIG. 6A depicts a flowchart for an exemplary process 600 of uploading an object from a client. FIG. 6B depicts a flowchart for an exemplary process 650 of removing an object deleted by a client. In one implementation, processes 600 and 650 may be performed by one of storage clusters 120. Processes 600 and 650 may be implemented in response to client (e.g., client 110) activities. For particular examples of processes 600 and 650 described below, reference may be made to storage cluster 120-1 of file system 140, where storage cluster 120-1 includes a cluster ID of “01.”
  • Referring to FIG. 6A, process 600 may begin when an uploaded file is received from a client (block 610). For example, storage cluster 120-1 may receive a new file from one of clients 110. The uploaded file may be stored (block 620) and a “Data” designator for the uploaded file may be written to the global index (block 630). For example, storage cluster 120-1 may store the uploaded file in a memory (e.g., storage device 350) and add a content handle for the object to the global index. Storage cluster 120-1 may also write a data record (e.g., “01:Location:Data”) to the replicated global index addressed by the content handle of the object.
  • Referring to FIG. 6B, process 650 may begin when a notice of a deleted file is received (block 660). For example, storage cluster 120-1 may receive an indication that one of clients 110 has deleted a file. A delete request may be initiated (block 670) and a “DeleteRequest” designator for the deleted file may be written to the global index (block 680). For example, storage cluster 120-1 may initiate a delete request to asynchronously remove the delete file from file system 140. Storage device 120-1 may also write a “DeleteRequest” record (e.g., “01:Location:DeleteReqeust”) to the replicated global index addressed by the content handle of the object.
  • FIG. 7 is a flowchart of an exemplary process 700 for performing de-duplication in a distributed multi-master data replication system (e.g., file system 140). In one implementation, process 700 may be performed by one of storage clusters 120. In another implementation, some or all of process 700 may be performed by another device or a group of devices, including or excluding storage cluster 120. Process 700 may be implemented periodically in each storage cluster 120 and may include a scan of all or a portion of the objects in the storage cluster 120. For particular examples of process 700 described below, reference may be made to storage clusters 120-1 and 120-2 of file system 140, where storage cluster 120-1 includes a cluster ID of “01” and storage cluster 120-2 includes a cluster ID of “02.”
  • As illustrated in FIG. 7, process 700 may begin with conducting a scan of the global index (block 710). For example, storage cluster 120-1 (using, e.g., de-duplication logic 420) may conduct a scan of all or a portion of the objects listed in the global index. The scan may identify, for example, multiple replicas and/or objects marked for deletion.
  • It may be determined if a delete request is encountered (block 720). For example, storage cluster 120-1 may encounter an object in the global index that includes a delete request designator (e.g., “02:Location:DeleteReqeust”) from another storage cluster (e.g., from storage cluster 120-2). If it is determined that a delete request is encountered (block 720-YES), then the delete request may be processed (block 730). For example, storage cluster 120-1 may process the delete request as described in more detail with respect to FIG. 8.
  • If it is determined that a delete request is not encountered (block 720-NO), then it may be determined if redundant replicas exist (block 740). Redundant replicas may be replicated objects in different locations that have no outstanding delete requests for the object. For example, storage cluster 120-1 may identify multiple replicas for the same object that correspond to a content handle for which storage cluster 120-1 is responsible. The multiple replicas may be stored, for example, in different storage clusters (e.g., storage cluster 120-1 and storage cluster 120-2) or in different locations within the same storage cluster.
  • If it is determined that redundant replicas exist (block 740-YES), then the redundant replicas(s) may be removed (block 750). For example, storage cluster 120-1 may remove the redundant replica(s) as described in more detail with respect to FIG. 9. If it is determined that redundant replicas do not exist (block 740-NO), then the process may return to block 710, where another scan of the global index may be conducted (block 710).
  • FIG. 8 illustrates exemplary operations associated with the processing of a delete request of block 730 of FIG. 7. A delete request may be encountered for an object (block 810). For example, a scan being conducted by storage cluster 120-1 may identify a content handle in the global index with a delete request designator previously written by storage cluster 120-1 to delete a replica in a certain storage cluster (e.g., “02:Location:DeleteRequest”). Assuming that storage cluster 120-1 is responsible for the content handle, storage cluster 120-1 may apply operations to determine if the replica can now be de-duplicated.
  • It may be determined if a de-duplication designator exists (block 820). For example, storage cluster 120-1 may review other records in the global index associated with the content handle to determine if a de-duplication designator exists (e.g., 02:Location:Deduped″). If it is determined that a de-duplication designator exists (block 820 YES), then the replica and the related records in the global index may be de-duplicated (block 830). For example, storage cluster 120-1 may initiate a delete request to delete the replica in storage cluster 120-2 (if any) and delete any records (e.g., “02:Location:*”, where “*” may be any designator) from the global index that relate to the content handle for the deleted replica.
  • If it is determined that a de-duplication designator does not exists (block 820-NO), then it may be determined if another live replica exists (block 840). For example, storage cluster 120-1 may review the content handle for the global index to determine whether another live replica exists for the object. The global index may include, for example, a data record for that content handle from another storage cluster (e.g., “03:Location:Data”).
  • If another live replica exists (block 840-YES), then the replica may be de-duplicated as described above with respect to block 830. If another live replica does not exist (block 840-NO), then it may be determined if all replicas have delete requests (block 850). For example, storage cluster 120-1 may review the content handle for the global index to determine whether all the replicas associated with the content handle have an outstanding delete request (e.g., “*:*:DeleteRequest”, where “*” may be any ClusterID and any location, respectively).
  • If it is determined that all replicas have delete requests (block 850-YES), then the replica may be de-duplicated as described above with respect to block 830. If it is determined that all replicas do not have delete requests (block 850-NO), then the object may be copied from a storage cluster that initiated a delete request to a different storage cluster and the global index may be updated (block 860). For example, in response to the record “02:Location:DeleteRequest,” storage cluster 120-1 may copy the object from storage cluster 120-2 to another storage cluster 120-3 for which there is a de-duplication record (e.g., “03:Location:Deduped”) and no outstanding delete request. Storage cluster 120-1 may delete the previous de-duplication record (e.g., “03:Location:Deduped”) associated with the replica and write a data designator (e.g., “03:Location:Data”) to the corresponding content handle of the object in the global index.
  • FIG. 9 illustrates exemplary operations associated with the removing of duplicate references of block 750 of FIG. 7. Multiple replicas with no delete requests may be identified (block 910). For example, storage cluster 120-1 may review the global index and identify two or more replicas that have no outstanding delete requests corresponding to a content handle for which storage cluster 120-01 is responsible.
  • Criteria to determine replica(s) to be de-duplicated may be applied (block 920). For example, storage cluster 120-1 may apply criteria to de-duplicate the redundant replica that may be stored within storage cluster 120-1. The criteria to de-duplicate redundant replicas may be based on a variety of factors, such as geographic proximity of the replicas, available storage capacity at a storage cluster, or other factors. Storage cluster 120-1 (e.g., using de-duplication logic 420) may apply the criteria to the two or more replicas that have no outstanding delete requests identified above. In some implementations, multiple replicas may be identified to be de-duplicated. In other implementations, storage cluster 120-1 may leave more than one live replica (e.g., a replica not marked for de-duplication).
  • The global index may be updated to designate de-duplicated replica(s) as “Deduped” (block 930). For example, for each de-duplicated replica, storage cluster 120-1 may delete the previous data record (e.g., “02:Location:Data”) associated with the replica and write a de-duplication designator (e.g., “02:Location:Deduped”) to the corresponding content handle in the global index.
  • De-duplication of the redundant replicas may be accomplished using de-duplication messages that are replicated as a part of the global index. The replicas marked for de-duplication may be stored within storage cluster 120-1 or within another storage cluster (e.g., storage cluster 120-2, 120-3, 120-4, etc.). In one implementation, storage cluster 120-1 may delete locally-stored replicas and the corresponding “01:Location:Data” record from the global index and add “01:Location:Deduped” to the global index. Storage cluster 120-1 may also initiate delete messages, using the replicated global index, to delete replicas stored in other clusters.
  • FIG. 10 provides a flowchart of an exemplary process 1000 for optimizing bandwidth consumption and reducing latency in a distributed multi-master data replication system (e.g., file system 140). In one implementation, process 1000 may be performed by one of storage clusters 120. In another implementation, some or all of process 1000 may be performed by another device or group of devices, including or excluding storage cluster 120. For particular examples of process 1000 described below, reference may be made to storage cluster 120-1 of file system 140, where the storage cluster 120-1 includes a cluster ID of “01.”
  • As illustrated in FIG. 1000, process 1000 may begin with receiving a request for an object (block 1010). For example, storage cluster 120-1 may receive a request from a client (e.g., client 110-1) to obtain an object.
  • Object locations may be looked up in the global index (block 1020). For example, storage cluster 120-1 may look up the replica location(s) for the object in the replicated global index using the content handle of the object.
  • The “best” replica location may be identified (block 1030). For example, assuming that more than one replica is available, storage cluster 120-1 may determine the “best” replica to retrieve to minimize network resources. For example, the “best” replica may be the replica that has the closest geographic location to storage cluster 120-1. In other implementations, the “best” replica may be based on a combination of available network connectivity, geographic location, and/or other criteria. Thus, in some implementations, the “best” replica for the object may be stored locally within storage cluster 120-1.
  • The object may be retrieved from the identified location (block 1040). For example, storage cluster 120-1 may request the “best” replica from the closest available storage cluster and receive the replica to satisfy the client request. Storage cluster 120-1 may then send the replica to the client.
  • Examples
  • FIG. 11 provides a portion 1100 of an exemplary global index according to an implementation described herein. The index may include, among other information, a content handle column 1110 and a De-duplication designation record column 1120. Assume, in exemplary index portion 1100, a distributed multi-master data replication system includes three storage clusters, XX, YY, and ZZ. A de-duplication algorithm may run periodically in each of storage clusters XX, YY, and ZZ and may scan all or a portion of the global index. Also, records (e.g., Data, DeleteRequest, and Deduped) may be written by one of storage clusters XX, YY, or ZZ to the global index associated with a particular object content handle. Modifications to the global index may be replicated to all other participating clusters (e.g., the remaining of storage clusters XX, YY, and ZZ).
  • As shown in FIG. 11, index portion 1100 includes content handles and associated delete designation records for four objects. “Handle11” has records indicating replicas are stored at storage cluster XX (“XX:Location01:Data”) and storage cluster YY (“YY:Location01:Data”), respectively. “Handle21” has a record indicating a replica is stored at storage cluster XX (“XX:Location02:Data”) and another replica at storage cluster YY has an ongoing delete request (“YY:Location:02:DeleteRequest”). “Handle31” has records indicating replicas are stored at storage cluster YY (“XX:Location03:Data”) and storage cluster ZZ (“ZZ:Location01:Data”), respectively. “Handle31” also has two records indicating the replicas have ongoing delete requests at storage cluster YY (“YY:Location03:DeleteRequest”) and storage cluster ZZ (“ZZ:Location01:DeleteRequest”). “Handle41” has records indicating a replica is stored at storage cluster YY (“XX:Location04:Data”) and a record indicating the replica with an ongoing delete request at storage cluster YY (“YY:Location04:DeleteRequest”). Handle41 also has one record indicating de-duplication of a replica has occurred (“ZZ:Location02:Deduped”). The de-duplication algorithm used by the storage clusters can operate using guidelines consistent with the principles described herein. Assume storage cluster XX is assigned responsibility for the portion of the global index including “Handle11,” “Handle21,” “Handle31,” and “Handle41.”
  • When an object is fully uploaded in a storage cluster, the storage cluster may write a data record (e.g., “ClusterID:Location:Data”) to the replicated global index addressed by the content handle of the object. For example, “XX:Location01:Data” and “YY:Location01:Data” illustrate data records for replicas of “Handle11.” Also, “XX:Location02:Data” illustrates a data record for a replica of “Handle21.” Similar data records can be seen for “Handle31” and “Handle 41.”
  • When an object is requested in a storage cluster, the storage cluster may look up the replica locations in the replicated global index using the content handle of the object and fetch the replica from the “best” (e.g., closest) cluster. For example, assuming an object corresponding to “Handle11” is requested at storage cluster ZZ and that storage cluster YY is closer to storage cluster ZZ than is storage cluster XX, storage cluster ZZ may request the object replica corresponding to “Handle11” from storage cluster YY.
  • When an object is deleted in a storage cluster, the storage cluster may write “ClusterID:Location:DeleteRequest” to the replicated global index addressed by the content handle of the object. For example, “YY:Location02:DeleteRequest” illustrates a record for a deleted replica of “Handle21” in storage cluster YY. Similarly, “YY:Location03:DeleteRequest” and “ZZ:Location:01:DeleteRequest” illustrate records for deleted replicas of “Handle31” for storage clusters YY and ZZ, respectively.
  • If the scan in a storage cluster encounters multiple replicas that have no outstanding delete requests corresponding to a content handle the storage cluster is responsible for, the storage cluster may delete redundant replicas of the object (possibly leaving more than one live replica). For each deleted replica in another storage cluster, the storage cluster may delete the data record and write a de-duplication record. For example, the scan in storage cluster XX may identify that “Handle11” has records indicating replicas are stored at storage cluster XX (“XX:Location01:Data”) and storage cluster YY (“YY:Location01:Data”), respectively. Based on criteria provided for removing redundant references, storage cluster XX may initiate deletion of the replica at storage cluster YY. Storage cluster XX may delete the record “YY:Location01:Data” shown in FIG. 11 and write “YY:Location01:Deduped” instead.
  • If the scan in storage cluster XX encounters a delete request (e.g., “ClusterID:Location:DeleteRequest”) for a replica in another storage cluster (e.g., storage cluster YY or ZZ) corresponding to a content handle that storage cluster XX is responsible for, storage cluster XX may apply the following analysis. If there is a “Deduped” record for the same storage cluster and location as the delete request, if there exists another live replica of the object, or if all replicas have outstanding delete requests, the storage cluster XX can delete the replica of the object in storage cluster YY or ZZ (if any) and delete the records “YY:Location:*” or “ZZ:Location:*.” For example, the replica for “Handle21” in storage cluster YY and the record “YY:Location02:DeleteRequest” may be deleted by storage cluster XX since another live object (indicated by the record “XX:Location02:Data”) exists. Similarly, the replica for “Handle31” in storage cluster YY and the record “YY:Location:03:DeleteRequest” may be deleted by storage cluster XX since both replicas in storage cluster YY and storage cluster ZZ have outstanding delete requests.
  • If storage cluster XX cannot delete the replica of the object in storage cluster YY or ZZ (e.g., there is not a “Deduped” record or another live replica of the object, and all replicas do not have outstanding delete requests), storage cluster XX can copy the object from YY or ZZ to another storage cluster for which there is a de-duplication record and no outstanding delete request, deleting the de-duplication record and writing a data record. For example, the replica for “Handle41” in storage cluster YY (“YY:Location04:DeleteRequest”) may trigger storage cluster XX to copy the object associated with “Handle41” to storage cluster ZZ. Storage cluster XX may update the global index to change “ZZ:Location02:Deduped” to “ZZ:Location02:Data.”
  • The correctness of the algorithm is straightforward as all deletion operations on the object are performed only by the scan process in the storage cluster responsible for its content handle. The algorithm also transparently deals with multiple object replicas in the same cluster that have different locations (e.g. XX:Location1 and XX:Location2).
  • CONCLUSION
  • Systems and/or methods described herein may store a global index of objects in a distributed data replication system and replicate the global index and some of the objects throughout the distributed data replication system. A storage cluster may be assigned as the responsible entity for de-duplication within a particular subset of the global index. The storage cluster may conduct a scan of the subset of the global index and identify redundant replicas based on the scan. The storage cluster may de-duplicate the redundant replicas stored locally or in a remote storage cluster.
  • The foregoing description of implementations provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention.
  • For example, in another implementation a synchronous version of the de-duplication algorithm may be used in which different storage clusters communicate directly rather than using the replication layer within a distributed data replication system.
  • Also, while series of blocks have been described with regard to FIGS. 6A-10, the order of the blocks may be modified in other implementations. Further, non-dependent blocks may be performed in parallel.
  • It will be apparent that embodiments, as described herein, may be implemented in many different forms of software, firmware, and hardware in the implementations illustrated in the figures. The actual software code or specialized control hardware used to implement embodiments described herein is not limiting of the invention. Thus, the operation and behavior of the embodiments were described without reference to the specific software code-it being understood that software and control hardware may be designed to implement the embodiments based on the description herein.
  • Further, certain implementations described herein may be implemented as “logic” or a “component” that performs one or more functions. This logic or component may include hardware, such as a processor, microprocessor, an application specific integrated circuit or a field programmable gate array, or a combination of hardware and software (e.g., software executed by a processor).
  • It should be emphasized that the term “comprises” and/or “comprising” when used in this specification is taken to specify the presence of stated features, integers, steps, or components, but does not preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.
  • Even though particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of the invention. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification.
  • No element, act, or instruction used in the description of the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. Further, the phrase “based on,” as used herein is intended to mean “based, at least in part, on” unless explicitly stated otherwise.

Claims (23)

1. A method performed by a device of a plurality of devices in a distributed data replication system, the method comprising:
storing an index of objects in the distributed data replication system, the index being replicated while the replicas of objects are stored locally by the plurality of devices in the distributed data replication system;
conducting a scan of at least a portion of the index;
identifying a redundant replica of the at least one of the objects based on the scan of the index; and
de-duplicating the redundant replica by writing a de-duplication record to a portion of the index.
2. The method of claim 1, where changes to the index are replicated using an underlying replication layer of the distributed data replication system.
3. The method of claim 1, where the device is the only device responsible for reducing redundant replicas for objects within a particular subset of the index.
4. The method of claim 1, where the index is a content addressable index including a content handle for each object within the index.
5. The method of claim 4, further comprising:
writing a data record or a delete request record to a portion of the index associated with the content handle of the object; and
replicating a portion of the index associated with the content handle of the object to one or more other devices of the plurality of devices.
6. The method of claim 1, further comprising:
receiving a request for another object included in the index;
determining, based on the index, a device selected out of the plurality of devices from which to retrieve a replica of the other object; and
retrieving the replica of the other object from the selected device.
7. The method of claim 1, where the identifying the redundant replica is further based on criteria to distinguish a preference between two or more replicas.
8. The method of claim 1, where the index is stored separately from the replica within the device.
9. A device of a plurality of devices in a distributed data replication system, the device comprising:
means for storing an index of objects in the distributed data replication system;
means for writing changes to the index to designate a status of a replica of one of the objects;
means for replicating the changes to the index to the plurality of devices in the distributed data replication system;
means for conducting a scan of at least a portion of the index;
means for identifying a redundant replica of the one of the objects based on the scan of the index; and
means for de-duplicating the redundant replica.
10. The system of claim 9, further comprising:
means for updating the index to reflect the status of a de-duplicated replica.
11. A system, comprising:
a memory to store instructions, a data store of objects, and an index of the objects in the data store; and
a processor to execute the instructions in the memory to:
identify a status of an object in the data store, the status relating to whether the object has a replica and whether a delete request is associated with the object,
write a de-duplication designation record to the index based on the status of the object,
replicate the index with the de-duplication designation record to one or more devices, and
receive, from one of the one or more devices, other de-duplication designation records associated with the object,
where the de-duplication designation record and the other de-duplication designation records provide a basis for deletion of one or more replicas of the object.
12. The system of claim 11, where the de-duplication designation record and the other de-duplication designation records are exchanged using a replication layer in a distributed multi-master data replication environment.
13. The system of claim 11, where the processor is further configured to:
conduct a scan of a portion of the index to identify redundant replicas of the object.
14. The system of claim 11, where the device is solely responsible for de-duplication of a subset of objects within the index.
15. The system of claim 11, where the de-duplication designation record comprises:
a cluster identification for the device storing the object;
a location identification for the storage location of the object within the device; and
a designation section for the de-duplication status of the object.
16. The system of claim 15, where the designation section includes one of:
a data designation to indicate a live replica of the object;
a delete request designation to indicate an ongoing delete request for a replica, and
a de-duplication designator to indicate a replica that has been selected for de-duplication.
17. A method performed by one or more devices, comprising:
storing an index of objects in multiple devices within a distributed data replication system and replicating the index throughout the distributed data replication system while storing the objects locally, where each device is responsible for de-duplication of the objects within a particular subset of the index;
conducting a scan of each of the subsets of the index to identify redundant replicas based on the scan;
de-duplicating the redundant replicas; and
automatically copying an object from a device with a replica having an ongoing delete request to a device with a replica having been previously de-duplicated.
18. The method of claim 17, further comprising:
writing a change to a portion of the index associated with one of the objects, the change including a data record, a delete request record, or a de-duplication record for a replica of the object.
19. The method of claim 17, further comprising:
replicating the change to the index using an underlying replication layer of the distributed data replication system.
20. The method of claim 17, where the index is a content addressable index including a content handle for each object within the index.
21. A computer-readable memory comprising computer-executable instructions, the computer-readable memory comprising:
one or more instructions to conduct a scan of a portion of an index of objects in a distributed data replication system;
one or more instructions to identify a redundant replica of one of the objects based on the scan of the portion of the index; and
one or more instructions to de-duplicate the redundant replica.
22. The computer-readable memory of claim 21, further comprising:
one or more instructions to write a change to a portion of the index associated with the redundant replica, the change including a de-duplication record for the redundant replica.
23. The computer-readable memory of claim 21, further comprising:
one or more instructions to receive a request for another object included in the index;
one or more instructions to select, based on the index, storage cluster from which to retrieve a replica of the other object; and
one or more instructions to retrieve the replica of the other object from the selected storage cluster.
US12/644,693 2008-12-22 2009-12-22 Asynchronous distributed de-duplication for replicated content addressable storage clusters Active 2032-02-09 US8712974B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US13985708P true 2008-12-22 2008-12-22
US12/644,693 US8712974B2 (en) 2008-12-22 2009-12-22 Asynchronous distributed de-duplication for replicated content addressable storage clusters

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US12/644,693 US8712974B2 (en) 2008-12-22 2009-12-22 Asynchronous distributed de-duplication for replicated content addressable storage clusters
US14/265,298 US20140236888A1 (en) 2008-12-22 2014-04-29 Asynchronous Distributed De-Duplication for Replicated Content Addressable Storage Clusters
US14/995,171 US10291699B2 (en) 2008-12-22 2016-01-13 Asynchronous distributed de-duplication for replicated content addressable storage clusters

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US14/265,298 Continuation US20140236888A1 (en) 2008-12-22 2014-04-29 Asynchronous Distributed De-Duplication for Replicated Content Addressable Storage Clusters

Publications (2)

Publication Number Publication Date
US20100161554A1 true US20100161554A1 (en) 2010-06-24
US8712974B2 US8712974B2 (en) 2014-04-29

Family

ID=41719028

Family Applications (3)

Application Number Title Priority Date Filing Date
US12/644,693 Active 2032-02-09 US8712974B2 (en) 2008-12-22 2009-12-22 Asynchronous distributed de-duplication for replicated content addressable storage clusters
US14/265,298 Abandoned US20140236888A1 (en) 2008-12-22 2014-04-29 Asynchronous Distributed De-Duplication for Replicated Content Addressable Storage Clusters
US14/995,171 Active 2031-04-23 US10291699B2 (en) 2008-12-22 2016-01-13 Asynchronous distributed de-duplication for replicated content addressable storage clusters

Family Applications After (2)

Application Number Title Priority Date Filing Date
US14/265,298 Abandoned US20140236888A1 (en) 2008-12-22 2014-04-29 Asynchronous Distributed De-Duplication for Replicated Content Addressable Storage Clusters
US14/995,171 Active 2031-04-23 US10291699B2 (en) 2008-12-22 2016-01-13 Asynchronous distributed de-duplication for replicated content addressable storage clusters

Country Status (9)

Country Link
US (3) US8712974B2 (en)
EP (1) EP2368199B1 (en)
JP (2) JP5579195B2 (en)
CN (2) CN104166673B (en)
AU (1) AU2009330073B2 (en)
BR (1) BRPI0922990A2 (en)
CA (1) CA2747746C (en)
DE (1) DE202009019139U1 (en)
WO (1) WO2010075407A1 (en)

Cited By (116)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070250519A1 (en) * 2006-04-25 2007-10-25 Fineberg Samuel A Distributed differential store with non-distributed objects and compression-enhancing data-object routing
US20080229037A1 (en) * 2006-12-04 2008-09-18 Alan Bunte Systems and methods for creating copies of data, such as archive copies
US20080243957A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
US20090319585A1 (en) * 2008-06-24 2009-12-24 Parag Gokhale Application-aware and remote single instance data management
US20100169287A1 (en) * 2008-11-26 2010-07-01 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US20100250549A1 (en) * 2009-03-30 2010-09-30 Muller Marcus S Storing a variable number of instances of data objects
US20100281077A1 (en) * 2009-04-30 2010-11-04 Mark David Lillibridge Batching requests for accessing differential data stores
US20100299490A1 (en) * 2009-05-22 2010-11-25 Attarde Deepak R Block-level single instancing
US20110060759A1 (en) * 2009-09-08 2011-03-10 International Business Machines Corporation Data De-Duplication in a Distributed Network
US8166263B2 (en) 2008-07-03 2012-04-24 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US20120191710A1 (en) * 2008-04-09 2012-07-26 John Howe Directed placement of data in a redundant data storage system
US20130073286A1 (en) * 2011-09-20 2013-03-21 Apple Inc. Consolidating Speech Recognition Results
US8433681B2 (en) 2011-05-12 2013-04-30 Dell Products L.P. System and method for managing replication in an object storage system
US20130144845A1 (en) * 2011-12-02 2013-06-06 International Business Machines Corporation Removal of data remanence in deduplicated storage clouds
US20130151483A1 (en) * 2011-12-07 2013-06-13 Quantum Corporation Adaptive experience based De-duplication
US20130151484A1 (en) * 2011-12-08 2013-06-13 Empire Technology Development, Llc. Storage discounts for allowing cross-user deduplication
US8468138B1 (en) 2011-12-02 2013-06-18 International Business Machines Corporation Managing redundant immutable files using deduplication in storage clouds
US8504595B2 (en) * 2011-12-07 2013-08-06 Jeffrey Tofano De-duplication for a global coherent de-duplication repository
US20140067775A1 (en) * 2012-09-05 2014-03-06 salesforce.com,inc System, method and computer program product for conditionally performing de-duping on data
US8732124B1 (en) * 2012-03-26 2014-05-20 Emc Corporation Multisite replication with coordinated cycle switching
US8892446B2 (en) 2010-01-18 2014-11-18 Apple Inc. Service orchestration for intelligent automated assistant
US8935492B2 (en) 2010-09-30 2015-01-13 Commvault Systems, Inc. Archiving data objects using secondary copies
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US8984028B2 (en) 2012-05-29 2015-03-17 Recommind, Inc. Systems and methods for storing data and eliminating redundancy
US9015181B2 (en) 2008-09-26 2015-04-21 Commvault Systems, Inc. Systems and methods for managing single instancing data
US9020890B2 (en) 2012-03-30 2015-04-28 Commvault Systems, Inc. Smart archiving and data previewing for mobile devices
US9086819B2 (en) * 2012-07-25 2015-07-21 Anoosmar Technologies Private Limited System and method for combining deduplication and encryption of data
US9098495B2 (en) 2008-06-24 2015-08-04 Commvault Systems, Inc. Application-aware and remote single instance data management
US9135016B1 (en) * 2010-05-14 2015-09-15 Emc Corporation Methods and apparatus for managing replication of objects in a storage environment
US9141621B2 (en) 2009-04-30 2015-09-22 Hewlett-Packard Development Company, L.P. Copying a differential data store into temporary storage media in response to a request
US9190062B2 (en) 2010-02-25 2015-11-17 Apple Inc. User profiling for voice input processing
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US9300784B2 (en) 2013-06-13 2016-03-29 Apple Inc. System and method for emergency calls initiated by voice command
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US9430490B1 (en) * 2014-03-28 2016-08-30 Formation Data Systems, Inc. Multi-tenant secure data deduplication using data association tables
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US9535906B2 (en) 2008-07-31 2017-01-03 Apple Inc. Mobile device having human language translation capability with positional feedback
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US9620104B2 (en) 2013-06-07 2017-04-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9626955B2 (en) 2008-04-05 2017-04-18 Apple Inc. Intelligent text-to-speech conversion
US9633674B2 (en) 2013-06-07 2017-04-25 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US9633022B2 (en) 2012-12-28 2017-04-25 Commvault Systems, Inc. Backup and restoration for a deduplicated file system
US20170124148A1 (en) * 2015-10-29 2017-05-04 International Business Machines Corporation Index table based routing for query resource optimization
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US9646614B2 (en) 2000-03-16 2017-05-09 Apple Inc. Fast, language-independent method for user authentication by voice
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US9697822B1 (en) 2013-03-15 2017-07-04 Apple Inc. System and method for updating an adaptive speech recognition model
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US9800575B1 (en) * 2014-09-24 2017-10-24 Ebay Inc. Assigning storage responsibility in a distributed data storage system with replication
US9798393B2 (en) 2011-08-29 2017-10-24 Apple Inc. Text correction processing
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9922642B2 (en) 2013-03-15 2018-03-20 Apple Inc. Training an at least partial voice command system
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9953088B2 (en) 2012-05-14 2018-04-24 Apple Inc. Crowd sourcing information to fulfill user requests
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US9966068B2 (en) 2013-06-08 2018-05-08 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US9971774B2 (en) 2012-09-19 2018-05-15 Apple Inc. Voice-based media searching
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US10079014B2 (en) 2012-06-08 2018-09-18 Apple Inc. Name recognition system
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10089072B2 (en) 2016-06-11 2018-10-02 Apple Inc. Intelligent device arbitration and control
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US10169363B2 (en) 2014-09-04 2019-01-01 International Business Machines Corporation Storing data in a distributed file system
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US10185542B2 (en) 2013-06-09 2019-01-22 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10199051B2 (en) 2013-02-07 2019-02-05 Apple Inc. Voice trigger for a digital assistant
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
US10269345B2 (en) 2016-06-11 2019-04-23 Apple Inc. Intelligent task discovery
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
US10297253B2 (en) 2016-06-11 2019-05-21 Apple Inc. Application integration with a digital assistant
US10303715B2 (en) 2017-05-16 2019-05-28 Apple Inc. Intelligent automated assistant for media exploration
US10311144B2 (en) 2017-05-16 2019-06-04 Apple Inc. Emoji word sense disambiguation
US10318871B2 (en) 2005-09-08 2019-06-11 Apple Inc. Method and apparatus for building an intelligent automated assistant
US10324897B2 (en) 2014-01-27 2019-06-18 Commvault Systems, Inc. Techniques for serving archived electronic mail
US10332518B2 (en) 2017-05-09 2019-06-25 Apple Inc. User interface for correcting recognition errors

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104166673B (en) * 2008-12-22 2017-09-19 谷歌公司 Asynchronous copy content addressable storage for distributed cluster deduplication
US9286052B1 (en) * 2011-09-15 2016-03-15 Emc Corporation Upgrading software on a pair of nodes in a clustered environment
CN103491124B (en) * 2012-06-14 2018-08-14 南京中兴软件有限责任公司 Species MMS and data processing method for distributed caching systems
US20160188397A1 (en) * 2013-07-29 2016-06-30 Hewlett-Packard Development Company, L.P. Integrity of frequently used de-duplication objects
CN104462396B (en) * 2014-12-10 2017-12-19 北京国双科技有限公司 String processing method and apparatus
US10324914B2 (en) 2015-05-20 2019-06-18 Commvalut Systems, Inc. Handling user queries against production and archive storage systems, such as for enterprise customers having large and/or numerous files
CN105187517A (en) * 2015-08-20 2015-12-23 浪潮(北京)电子信息产业有限公司 Metadata clustering method and metadata cluster

Citations (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5551027A (en) * 1993-01-07 1996-08-27 International Business Machines Corporation Multi-tiered indexing method for partitioned data
US6438562B1 (en) * 1999-08-24 2002-08-20 Oracle Corporation Parallel index maintenance
US20040122958A1 (en) * 2002-12-19 2004-06-24 International Business Machines Corporation Method and system for peer-to-peer authorization
US20050177603A1 (en) * 2004-02-06 2005-08-11 Availl, Inc. System and method for replicating files in a computer network
US20050193024A1 (en) * 2004-02-27 2005-09-01 Beyer Kevin S. Asynchronous peer-to-peer data replication
US20060015544A1 (en) * 2004-07-13 2006-01-19 Hitachi, Ltd. Incorporation: Japan Data management system
US20060047999A1 (en) * 2004-08-24 2006-03-02 Ron Passerini Generation and use of a time map for accessing a prior image of a storage device
US7200604B2 (en) * 2004-02-17 2007-04-03 Hewlett-Packard Development Company, L.P. Data de-duplication
US20080301087A1 (en) * 2007-05-30 2008-12-04 Red Hat, Inc. Index clustering for full text search engines
US20090144422A1 (en) * 2007-08-29 2009-06-04 Chatley Scott P Global load based file allocation among a plurality of geographically distributed storage nodes
US20090204636A1 (en) * 2008-02-11 2009-08-13 Microsoft Corporation Multimodal object de-duplication
US20090313312A1 (en) * 2008-06-11 2009-12-17 International Business Machines Corporation Method of Enhancing De-Duplication Impact by Preferential Selection of Master Copy to be Retained
US20100005151A1 (en) * 2008-07-02 2010-01-07 Parag Gokhale Distributed indexing system for data storage
US7669023B2 (en) * 2007-07-10 2010-02-23 Hitachi, Ltd. Power efficient storage with data de-duplication
US20100082558A1 (en) * 2008-10-01 2010-04-01 International Business Machines Corporation Policy-based sharing of redundant data across storage pools in a deduplicating system
US7747584B1 (en) * 2006-08-22 2010-06-29 Netapp, Inc. System and method for enabling de-duplication in a storage system architecture
US7783604B1 (en) * 2007-12-31 2010-08-24 Emc Corporation Data de-duplication and offsite SaaS backup and archiving
US7814499B2 (en) * 1996-07-01 2010-10-12 Microsoft Corporation Urgent replication facility
US7814149B1 (en) * 2008-09-29 2010-10-12 Symantec Operating Corporation Client side data deduplication
US7814074B2 (en) * 2008-03-14 2010-10-12 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US7822939B1 (en) * 2007-09-25 2010-10-26 Emc Corporation Data de-duplication using thin provisioning
US7840537B2 (en) * 2006-12-22 2010-11-23 Commvault Systems, Inc. System and method for storing redundant information
US20100325476A1 (en) * 2004-11-19 2010-12-23 Microsoft Corporation System and method for a distributed object store
US7870105B2 (en) * 2007-11-20 2011-01-11 Hitachi, Ltd. Methods and apparatus for deduplication in storage system
US7870409B2 (en) * 2007-09-26 2011-01-11 Hitachi, Ltd. Power efficient data storage with data de-duplication
US7873809B2 (en) * 2007-03-29 2011-01-18 Hitachi, Ltd. Method and apparatus for de-duplication after mirror operation
US20110040728A1 (en) * 2009-08-11 2011-02-17 International Business Machines Corporation Replication of deduplicated data
US7908436B1 (en) * 2008-04-25 2011-03-15 Netapp, Inc. Deduplication of data on disk devices using low-latency random read memory
US7949622B2 (en) * 2007-12-13 2011-05-24 Yahoo! Inc. System and method for generating a classifier model for classifying web content
US7962706B2 (en) * 2008-02-14 2011-06-14 Quantum Corporation Methods and systems for improving read performance in data de-duplication storage
US7984022B2 (en) * 2008-04-18 2011-07-19 International Business Machines Corporation Space recovery with storage management coupled with a deduplicating storage system
US7984026B2 (en) * 2006-12-26 2011-07-19 Hitachi, Ltd. Storage system with improved de-duplication arrangement
US7996371B1 (en) * 2008-06-10 2011-08-09 Netapp, Inc. Combining context-aware and context-independent data deduplication for optimal space savings
US8108353B2 (en) * 2008-06-11 2012-01-31 International Business Machines Corporation Method and apparatus for block size optimization in de-duplication
US8135918B1 (en) * 2007-12-31 2012-03-13 Emc Corporation Data de-duplication for iSCSI
US8190835B1 (en) * 2007-12-31 2012-05-29 Emc Corporation Global de-duplication in shared architectures
US8548953B2 (en) * 2007-11-12 2013-10-01 F5 Networks, Inc. File deduplication using storage tiers

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU6500596A (en) * 1995-07-20 1997-02-18 Novell, Inc. Transaction log management in a disconnectable computer and network
US5897664A (en) * 1996-07-01 1999-04-27 Sun Microsystems, Inc. Multiprocessor system having mapping table in each node to map global physical addresses to local physical addresses of page copies
US5812773A (en) * 1996-07-12 1998-09-22 Microsoft Corporation System and method for the distribution of hierarchically structured data
CN1232914C (en) 2000-04-26 2005-12-21 计算机合作者思维公司 Method and apparatus for maintaining data integrity across distributed computer systems
JP4096147B2 (en) * 2000-10-25 2008-06-04 三菱電機株式会社 Replication method of overlapping arrangement replicated data in a distributed computer system
US20020161907A1 (en) * 2001-04-25 2002-10-31 Avery Moon Adaptive multi-protocol communications system
US6988124B2 (en) * 2001-06-06 2006-01-17 Microsoft Corporation Locating potentially identical objects across multiple computers based on stochastic partitioning of workload
US7573500B2 (en) * 2003-03-24 2009-08-11 Sensormatic Electronics Corporation System and method for communicating data in a video system
JP2006185041A (en) * 2004-12-27 2006-07-13 Matsushita Electric Ind Co Ltd Content distributed arrangement system, terminal, and operation method of content distributed arrangement system
US7584338B1 (en) * 2005-09-27 2009-09-01 Data Domain, Inc. Replication of deduplicated storage system
US7788223B2 (en) * 2005-12-05 2010-08-31 Microsoft Corporation Resource freshness and replication
JP4920979B2 (en) * 2006-01-25 2012-04-18 株式会社日立製作所 The storage device and control method thereof
US7921077B2 (en) * 2006-06-29 2011-04-05 Netapp, Inc. System and method for managing data deduplication of storage systems utilizing persistent consistency point images
US8635194B2 (en) * 2006-10-19 2014-01-21 Oracle International Corporation System and method for data compression
JP4945232B2 (en) * 2006-12-21 2012-06-06 株式会社日立製作所 Access control method, computer system, and object copying program
US8719375B2 (en) * 2007-03-22 2014-05-06 Microsoft Corporation Remote data access techniques for portable devices
US8768895B2 (en) * 2007-04-11 2014-07-01 Emc Corporation Subsegmenting for efficient storage, resemblance determination, and transmission
US8315984B2 (en) 2007-05-22 2012-11-20 Netapp, Inc. System and method for on-the-fly elimination of redundant data
US8793226B1 (en) * 2007-08-28 2014-07-29 Netapp, Inc. System and method for estimating duplicate data
US8346730B2 (en) * 2008-04-25 2013-01-01 Netapp. Inc. Deduplication of data on disk devices based on a threshold number of sequential blocks
US8645333B2 (en) * 2008-05-29 2014-02-04 International Business Machines Corporation Method and apparatus to minimize metadata in de-duplication
US8788466B2 (en) * 2008-08-05 2014-07-22 International Business Machines Corporation Efficient transfer of deduplicated data
CN104166673B (en) * 2008-12-22 2017-09-19 谷歌公司 Asynchronous copy content addressable storage for distributed cluster deduplication
US8868508B2 (en) * 2010-02-09 2014-10-21 Google Inc. Storage of data in a distributed storage system
AU2014265979A1 (en) * 2013-05-14 2015-12-10 Actifio, Inc. Efficient data replication and garbage collection predictions
US9928248B2 (en) * 2013-07-08 2018-03-27 International Business Machines Corporation Self-healing by hash-based deduplication
US9817865B2 (en) * 2015-12-07 2017-11-14 International Business Machines Corporation Direct lookup for identifying duplicate data in a data deduplication system

Patent Citations (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5551027A (en) * 1993-01-07 1996-08-27 International Business Machines Corporation Multi-tiered indexing method for partitioned data
US7814499B2 (en) * 1996-07-01 2010-10-12 Microsoft Corporation Urgent replication facility
US6438562B1 (en) * 1999-08-24 2002-08-20 Oracle Corporation Parallel index maintenance
US20040122958A1 (en) * 2002-12-19 2004-06-24 International Business Machines Corporation Method and system for peer-to-peer authorization
US20050177603A1 (en) * 2004-02-06 2005-08-11 Availl, Inc. System and method for replicating files in a computer network
US7200604B2 (en) * 2004-02-17 2007-04-03 Hewlett-Packard Development Company, L.P. Data de-duplication
US20050193024A1 (en) * 2004-02-27 2005-09-01 Beyer Kevin S. Asynchronous peer-to-peer data replication
US20060015544A1 (en) * 2004-07-13 2006-01-19 Hitachi, Ltd. Incorporation: Japan Data management system
US20060047999A1 (en) * 2004-08-24 2006-03-02 Ron Passerini Generation and use of a time map for accessing a prior image of a storage device
US20100325476A1 (en) * 2004-11-19 2010-12-23 Microsoft Corporation System and method for a distributed object store
US7747584B1 (en) * 2006-08-22 2010-06-29 Netapp, Inc. System and method for enabling de-duplication in a storage system architecture
US7840537B2 (en) * 2006-12-22 2010-11-23 Commvault Systems, Inc. System and method for storing redundant information
US7984026B2 (en) * 2006-12-26 2011-07-19 Hitachi, Ltd. Storage system with improved de-duplication arrangement
US7873809B2 (en) * 2007-03-29 2011-01-18 Hitachi, Ltd. Method and apparatus for de-duplication after mirror operation
US20080301087A1 (en) * 2007-05-30 2008-12-04 Red Hat, Inc. Index clustering for full text search engines
US7669023B2 (en) * 2007-07-10 2010-02-23 Hitachi, Ltd. Power efficient storage with data de-duplication
US20090144422A1 (en) * 2007-08-29 2009-06-04 Chatley Scott P Global load based file allocation among a plurality of geographically distributed storage nodes
US7822939B1 (en) * 2007-09-25 2010-10-26 Emc Corporation Data de-duplication using thin provisioning
US7870409B2 (en) * 2007-09-26 2011-01-11 Hitachi, Ltd. Power efficient data storage with data de-duplication
US8548953B2 (en) * 2007-11-12 2013-10-01 F5 Networks, Inc. File deduplication using storage tiers
US7870105B2 (en) * 2007-11-20 2011-01-11 Hitachi, Ltd. Methods and apparatus for deduplication in storage system
US7949622B2 (en) * 2007-12-13 2011-05-24 Yahoo! Inc. System and method for generating a classifier model for classifying web content
US7783604B1 (en) * 2007-12-31 2010-08-24 Emc Corporation Data de-duplication and offsite SaaS backup and archiving
US8190835B1 (en) * 2007-12-31 2012-05-29 Emc Corporation Global de-duplication in shared architectures
US8135918B1 (en) * 2007-12-31 2012-03-13 Emc Corporation Data de-duplication for iSCSI
US20090204636A1 (en) * 2008-02-11 2009-08-13 Microsoft Corporation Multimodal object de-duplication
US7962706B2 (en) * 2008-02-14 2011-06-14 Quantum Corporation Methods and systems for improving read performance in data de-duplication storage
US7814074B2 (en) * 2008-03-14 2010-10-12 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US7984022B2 (en) * 2008-04-18 2011-07-19 International Business Machines Corporation Space recovery with storage management coupled with a deduplicating storage system
US7908436B1 (en) * 2008-04-25 2011-03-15 Netapp, Inc. Deduplication of data on disk devices using low-latency random read memory
US7996371B1 (en) * 2008-06-10 2011-08-09 Netapp, Inc. Combining context-aware and context-independent data deduplication for optimal space savings
US20090313312A1 (en) * 2008-06-11 2009-12-17 International Business Machines Corporation Method of Enhancing De-Duplication Impact by Preferential Selection of Master Copy to be Retained
US8108353B2 (en) * 2008-06-11 2012-01-31 International Business Machines Corporation Method and apparatus for block size optimization in de-duplication
US20100005151A1 (en) * 2008-07-02 2010-01-07 Parag Gokhale Distributed indexing system for data storage
US7814149B1 (en) * 2008-09-29 2010-10-12 Symantec Operating Corporation Client side data deduplication
US20100082558A1 (en) * 2008-10-01 2010-04-01 International Business Machines Corporation Policy-based sharing of redundant data across storage pools in a deduplicating system
US20110040728A1 (en) * 2009-08-11 2011-02-17 International Business Machines Corporation Replication of deduplicated data

Cited By (167)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9646614B2 (en) 2000-03-16 2017-05-09 Apple Inc. Fast, language-independent method for user authentication by voice
US10318871B2 (en) 2005-09-08 2019-06-11 Apple Inc. Method and apparatus for building an intelligent automated assistant
US8447864B2 (en) 2006-04-25 2013-05-21 Hewlett-Packard Development Company, L.P. Distributed differential store with non-distributed objects and compression-enhancing data-object routing
US20070250519A1 (en) * 2006-04-25 2007-10-25 Fineberg Samuel A Distributed differential store with non-distributed objects and compression-enhancing data-object routing
US8190742B2 (en) 2006-04-25 2012-05-29 Hewlett-Packard Development Company, L.P. Distributed differential store with non-distributed objects and compression-enhancing data-object routing
US9117447B2 (en) 2006-09-08 2015-08-25 Apple Inc. Using event alert text as input to an automated assistant
US8942986B2 (en) 2006-09-08 2015-01-27 Apple Inc. Determining user intent based on ontologies of domains
US8930191B2 (en) 2006-09-08 2015-01-06 Apple Inc. Paraphrasing of user requests and results by automated digital assistant
US8909881B2 (en) 2006-11-28 2014-12-09 Commvault Systems, Inc. Systems and methods for creating copies of data, such as archive copies
US8140786B2 (en) 2006-12-04 2012-03-20 Commvault Systems, Inc. Systems and methods for creating copies of data, such as archive copies
US20080229037A1 (en) * 2006-12-04 2008-09-18 Alan Bunte Systems and methods for creating copies of data, such as archive copies
US8392677B2 (en) 2006-12-04 2013-03-05 Commvault Systems, Inc. Systems and methods for creating copies of data, such as archive copies
US7953706B2 (en) 2006-12-22 2011-05-31 Commvault Systems, Inc. System and method for storing redundant information
US8037028B2 (en) 2006-12-22 2011-10-11 Commvault Systems, Inc. System and method for storing redundant information
US8712969B2 (en) 2006-12-22 2014-04-29 Commvault Systems, Inc. System and method for storing redundant information
US20080243958A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
US20080243957A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
US8285683B2 (en) 2006-12-22 2012-10-09 Commvault Systems, Inc. System and method for storing redundant information
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US9865248B2 (en) 2008-04-05 2018-01-09 Apple Inc. Intelligent text-to-speech conversion
US9626955B2 (en) 2008-04-05 2017-04-18 Apple Inc. Intelligent text-to-speech conversion
US8504571B2 (en) * 2008-04-09 2013-08-06 Harmonic Inc. Directed placement of data in a redundant data storage system
US20120191710A1 (en) * 2008-04-09 2012-07-26 John Howe Directed placement of data in a redundant data storage system
US20090319585A1 (en) * 2008-06-24 2009-12-24 Parag Gokhale Application-aware and remote single instance data management
US8219524B2 (en) * 2008-06-24 2012-07-10 Commvault Systems, Inc. Application-aware and remote single instance data management
US9098495B2 (en) 2008-06-24 2015-08-04 Commvault Systems, Inc. Application-aware and remote single instance data management
US8612707B2 (en) 2008-07-03 2013-12-17 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8380957B2 (en) 2008-07-03 2013-02-19 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8166263B2 (en) 2008-07-03 2012-04-24 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8838923B2 (en) 2008-07-03 2014-09-16 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US9535906B2 (en) 2008-07-31 2017-01-03 Apple Inc. Mobile device having human language translation capability with positional feedback
US10108612B2 (en) 2008-07-31 2018-10-23 Apple Inc. Mobile device having human language translation capability with positional feedback
US9015181B2 (en) 2008-09-26 2015-04-21 Commvault Systems, Inc. Systems and methods for managing single instancing data
US9158787B2 (en) 2008-11-26 2015-10-13 Commvault Systems, Inc Systems and methods for byte-level or quasi byte-level single instancing
US8725687B2 (en) 2008-11-26 2014-05-13 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US20100169287A1 (en) * 2008-11-26 2010-07-01 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US8412677B2 (en) 2008-11-26 2013-04-02 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US9773025B2 (en) 2009-03-30 2017-09-26 Commvault Systems, Inc. Storing a variable number of instances of data objects
US20100250549A1 (en) * 2009-03-30 2010-09-30 Muller Marcus S Storing a variable number of instances of data objects
US8401996B2 (en) 2009-03-30 2013-03-19 Commvault Systems, Inc. Storing a variable number of instances of data objects
US20100281077A1 (en) * 2009-04-30 2010-11-04 Mark David Lillibridge Batching requests for accessing differential data stores
US9141621B2 (en) 2009-04-30 2015-09-22 Hewlett-Packard Development Company, L.P. Copying a differential data store into temporary storage media in response to a request
US9058117B2 (en) 2009-05-22 2015-06-16 Commvault Systems, Inc. Block-level single instancing
US8578120B2 (en) 2009-05-22 2013-11-05 Commvault Systems, Inc. Block-level single instancing
US20100299490A1 (en) * 2009-05-22 2010-11-25 Attarde Deepak R Block-level single instancing
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US10283110B2 (en) 2009-07-02 2019-05-07 Apple Inc. Methods and apparatuses for automatic speech recognition
US9116910B2 (en) 2009-09-08 2015-08-25 International Business Machines Corporation Data de-duplication in a distributed network
US20110060759A1 (en) * 2009-09-08 2011-03-10 International Business Machines Corporation Data De-Duplication in a Distributed Network
US8572137B2 (en) * 2009-09-08 2013-10-29 International Business Machines Corporation Data de-duplication in a distributed network
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US9548050B2 (en) 2010-01-18 2017-01-17 Apple Inc. Intelligent automated assistant
US8892446B2 (en) 2010-01-18 2014-11-18 Apple Inc. Service orchestration for intelligent automated assistant
US8903716B2 (en) 2010-01-18 2014-12-02 Apple Inc. Personalized vocabulary for digital assistant
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US9633660B2 (en) 2010-02-25 2017-04-25 Apple Inc. User profiling for voice input processing
US10049675B2 (en) 2010-02-25 2018-08-14 Apple Inc. User profiling for voice input processing
US9190062B2 (en) 2010-02-25 2015-11-17 Apple Inc. User profiling for voice input processing
US9135016B1 (en) * 2010-05-14 2015-09-15 Emc Corporation Methods and apparatus for managing replication of objects in a storage environment
US9262275B2 (en) 2010-09-30 2016-02-16 Commvault Systems, Inc. Archiving data objects using secondary copies
US8935492B2 (en) 2010-09-30 2015-01-13 Commvault Systems, Inc. Archiving data objects using secondary copies
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US10102359B2 (en) 2011-03-21 2018-10-16 Apple Inc. Device access using voice authentication
US8433681B2 (en) 2011-05-12 2013-04-30 Dell Products L.P. System and method for managing replication in an object storage system
US8930313B2 (en) 2011-05-12 2015-01-06 Dell Products L.P. System and method for managing replication in an object storage system
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US9798393B2 (en) 2011-08-29 2017-10-24 Apple Inc. Text correction processing
AU2012227212B2 (en) * 2011-09-20 2015-05-21 Apple Inc. Consolidating speech recognition results
US20130073286A1 (en) * 2011-09-20 2013-03-21 Apple Inc. Consolidating Speech Recognition Results
CN103077715A (en) * 2011-09-20 2013-05-01 苹果公司 Consolidating speech recognition results
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US8892521B2 (en) 2011-12-02 2014-11-18 International Business Machines Corporation Managing redundant immutable files using deduplication in storage clouds
US20130144845A1 (en) * 2011-12-02 2013-06-06 International Business Machines Corporation Removal of data remanence in deduplicated storage clouds
US9514145B2 (en) 2011-12-02 2016-12-06 International Business Machines Corporation Managing redundant immutable files using deduplication in storage clouds
US9158783B2 (en) 2011-12-02 2015-10-13 International Business Machines Corporation Managing redundant immutable files using deduplication in storage clouds
US8682868B2 (en) * 2011-12-02 2014-03-25 International Business Machines Corporation Removal of data remanence in deduplicated storage clouds
US8468138B1 (en) 2011-12-02 2013-06-18 International Business Machines Corporation Managing redundant immutable files using deduplication in storage clouds
US8504595B2 (en) * 2011-12-07 2013-08-06 Jeffrey Tofano De-duplication for a global coherent de-duplication repository
US20130151483A1 (en) * 2011-12-07 2013-06-13 Quantum Corporation Adaptive experience based De-duplication
US20130151484A1 (en) * 2011-12-08 2013-06-13 Empire Technology Development, Llc. Storage discounts for allowing cross-user deduplication
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US8732124B1 (en) * 2012-03-26 2014-05-20 Emc Corporation Multisite replication with coordinated cycle switching
US9020890B2 (en) 2012-03-30 2015-04-28 Commvault Systems, Inc. Smart archiving and data previewing for mobile devices
US9953088B2 (en) 2012-05-14 2018-04-24 Apple Inc. Crowd sourcing information to fulfill user requests
US8984028B2 (en) 2012-05-29 2015-03-17 Recommind, Inc. Systems and methods for storing data and eliminating redundancy
US10079014B2 (en) 2012-06-08 2018-09-18 Apple Inc. Name recognition system
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9086819B2 (en) * 2012-07-25 2015-07-21 Anoosmar Technologies Private Limited System and method for combining deduplication and encryption of data
US20140067775A1 (en) * 2012-09-05 2014-03-06 salesforce.com,inc System, method and computer program product for conditionally performing de-duping on data
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9971774B2 (en) 2012-09-19 2018-05-15 Apple Inc. Voice-based media searching
US9633022B2 (en) 2012-12-28 2017-04-25 Commvault Systems, Inc. Backup and restoration for a deduplicated file system
US10199051B2 (en) 2013-02-07 2019-02-05 Apple Inc. Voice trigger for a digital assistant
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
US9697822B1 (en) 2013-03-15 2017-07-04 Apple Inc. System and method for updating an adaptive speech recognition model
US9922642B2 (en) 2013-03-15 2018-03-20 Apple Inc. Training an at least partial voice command system
US9633674B2 (en) 2013-06-07 2017-04-25 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
US9966060B2 (en) 2013-06-07 2018-05-08 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9620104B2 (en) 2013-06-07 2017-04-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9966068B2 (en) 2013-06-08 2018-05-08 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US10185542B2 (en) 2013-06-09 2019-01-22 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US9300784B2 (en) 2013-06-13 2016-03-29 Apple Inc. System and method for emergency calls initiated by voice command
US10324897B2 (en) 2014-01-27 2019-06-18 Commvault Systems, Inc. Techniques for serving archived electronic mail
US9430490B1 (en) * 2014-03-28 2016-08-30 Formation Data Systems, Inc. Multi-tenant secure data deduplication using data association tables
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US10083690B2 (en) 2014-05-30 2018-09-25 Apple Inc. Better resolution when referencing to concepts
US10169329B2 (en) 2014-05-30 2019-01-01 Apple Inc. Exemplar-based natural language processing
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9668024B2 (en) 2014-06-30 2017-05-30 Apple Inc. Intelligent automated assistant for TV user interactions
US10169363B2 (en) 2014-09-04 2019-01-01 International Business Machines Corporation Storing data in a distributed file system
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US9800575B1 (en) * 2014-09-24 2017-10-24 Ebay Inc. Assigning storage responsibility in a distributed data storage system with replication
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US9986419B2 (en) 2014-09-30 2018-05-29 Apple Inc. Social reminders
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US10311871B2 (en) 2015-03-08 2019-06-04 Apple Inc. Competing devices responding to voice triggers
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US20170124148A1 (en) * 2015-10-29 2017-05-04 International Business Machines Corporation Index table based routing for query resource optimization
US10042885B2 (en) * 2015-10-29 2018-08-07 International Business Machines Corporation Index table based routing for query resource optimization
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10269345B2 (en) 2016-06-11 2019-04-23 Apple Inc. Intelligent task discovery
US10089072B2 (en) 2016-06-11 2018-10-02 Apple Inc. Intelligent device arbitration and control
US10297253B2 (en) 2016-06-11 2019-05-21 Apple Inc. Application integration with a digital assistant
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10332518B2 (en) 2017-05-09 2019-06-25 Apple Inc. User interface for correcting recognition errors
US10303715B2 (en) 2017-05-16 2019-05-28 Apple Inc. Intelligent automated assistant for media exploration
US10311144B2 (en) 2017-05-16 2019-06-04 Apple Inc. Emoji word sense disambiguation

Also Published As

Publication number Publication date
WO2010075407A1 (en) 2010-07-01
DE202009019139U1 (en) 2017-01-20
CN104166673B (en) 2017-09-19
US10291699B2 (en) 2019-05-14
CA2747746C (en) 2015-08-11
CN102317938B (en) 2014-07-30
CN102317938A (en) 2012-01-11
AU2009330073B2 (en) 2012-11-15
JP2012513640A (en) 2012-06-14
AU2009330073A1 (en) 2011-07-07
US8712974B2 (en) 2014-04-29
US20160134696A1 (en) 2016-05-12
EP2368199B1 (en) 2018-10-31
JP2014139824A (en) 2014-07-31
US20140236888A1 (en) 2014-08-21
CA2747746A1 (en) 2010-07-01
EP2368199A1 (en) 2011-09-28
BRPI0922990A2 (en) 2016-01-26
JP5579195B2 (en) 2014-08-27
JP5902222B2 (en) 2016-04-13
CN104166673A (en) 2014-11-26

Similar Documents

Publication Publication Date Title
EP2904525B1 (en) Optimizing data block size for deduplication
US8886602B2 (en) Location assignment daemon (LAD) for a distributed storage system
CA2809224C (en) Storage system
US8504571B2 (en) Directed placement of data in a redundant data storage system
US9542413B2 (en) Stored data deduplication method, stored data deduplication apparatus, and deduplication program
JP5445682B2 (en) Storage system
US10114564B2 (en) Management of virtual machine snapshots
CN1307580C (en) Efficient management of large files
Bhagwat et al. Extreme binning: Scalable, parallel deduplication for chunk-based file backup
US8677091B2 (en) Writing data and storage system specific metadata to network attached storage device
US8370315B1 (en) System and method for high performance deduplication indexing
JP5539683B2 (en) Expandable secondary storage system and method
CN102597983B (en) Metadata virtual hard disk drive and a differential backup virtual hard drive
US8370593B2 (en) Method and apparatus to manage groups for deduplication
US8880824B1 (en) Global de-duplication in shared architectures
Shvachko et al. The hadoop distributed file system.
US10157184B2 (en) Data previewing before recalling large data files
US8683156B2 (en) Format-preserving deduplication of data
US8694469B2 (en) Cloud synthetic backups
US9489132B2 (en) Utilizing unmapped and unknown states in a replicated storage system
CN102317939B (en) Asynchronous distributed garbage collection for replicated storage clusters
US8433689B1 (en) Systems and methods for restoring deduplicated data
US20160098191A1 (en) Optimizing replication by distinguishing user and system write activity
US9239843B2 (en) Scalable de-duplication for storage systems
US20070220029A1 (en) System and method for hierarchical storage management using shadow volumes

Legal Events

Date Code Title Description
AS Assignment

Owner name: GOOGLE, INC.,CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DATUASHVILI, GIA;KESSELMAN, ALEXANDER;DROBYCHEV, ALEXANDRE;REEL/FRAME:023961/0486

Effective date: 20100120

Owner name: GOOGLE, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DATUASHVILI, GIA;KESSELMAN, ALEXANDER;DROBYCHEV, ALEXANDRE;REEL/FRAME:023961/0486

Effective date: 20100120

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044277/0001

Effective date: 20170929

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551)

Year of fee payment: 4