WO2010075407A1 - Asynchronous distributed de-duplication for replicated content addressable storage clusters - Google Patents
Asynchronous distributed de-duplication for replicated content addressable storage clusters Download PDFInfo
- Publication number
- WO2010075407A1 WO2010075407A1 PCT/US2009/069234 US2009069234W WO2010075407A1 WO 2010075407 A1 WO2010075407 A1 WO 2010075407A1 US 2009069234 W US2009069234 W US 2009069234W WO 2010075407 A1 WO2010075407 A1 WO 2010075407A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- index
- replica
- duplication
- objects
- storage cluster
- Prior art date
Links
- 238000000034 method Methods 0.000 claims abstract description 57
- 230000010076 replication Effects 0.000 claims abstract description 48
- 230000008859 change Effects 0.000 claims description 7
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 230000003362 replicative effect Effects 0.000 claims description 6
- 230000008569 process Effects 0.000 description 31
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 7
- 230000007246 mechanism Effects 0.000 description 4
- 238000012552 review Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 239000004020 conductor Substances 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000001066 destructive effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/184—Distributed file systems implemented as replicated file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Definitions
- the enterprise computing landscape has undergone a fundamental shift in storage architectures in that central-service architecture has given way to distributed storage clusters.
- storage clusters built from commodity computers can deliver high performance, availability and scalability for new data-intensive applications at a fraction of the cost compared to monolithic disk arrays.
- the data is replicated across multiple geographical locations, thereby increasing availability and reducing network distance from clients.
- Data de-duplication can identify duplicate objects and reduce required storage space by removing duplicates.
- data de-duplication is becoming increasingly important for a storage industry and is being driven by the needs of large-scale systems that can contain many duplicates.
- a method may be performed by a device of a group of devices in a distributed data replication system.
- the method may include storing an index of objects in the distributed data replication system, the index being replicated while the replicas of objects are stored locally by the plurality of devices in the distributed data replication system.
- the method may also include conducting a scan of at least a portion of the index and identifying a redundant replica of the at least one of the objects based on the scan of the index.
- the method may further include de- duplicating the redundant replica by writing a de-duplication record to a portion of the index.
- a device of a group of devices in a distributed data replication system, may include means for storing an index of objects in the distributed data replication system; means for writing changes to the index to designate a status of a replica of one of the objects; means for replicating the changes to the index to the plurality of devices in the distributed data replication system; means for conducting a scan of at least a portion of the index; means for identifying a redundant replica of the one of the objects based on the scan of the index; and means for de-duplicating the redundant replica.
- a system may include a memory to store instructions, a data store of objects and an index of the objects in the data store; and a processor.
- the processor may execute instructions in the memory to identify a status of an object in the data store, the status relating to whether the object has a replica and whether a delete request is associated with the object, write a de-duplication designation record to the index based on the status of the object, replicate the index with the de-duplication designation record to one or more devices, and receive, from one of the one or more devices, other de-duplication designation records associated with the object, where the de-duplication designation record and the other de-duplication designation records provide a basis for deletion of one or more replicas of the object.
- a method performed by one or more devices may include storing an index of objects in multiple devices within a distributed data replication system and replicating the index throughout the distributed data replication system while storing the objects locally, where each device is responsible for de-duplication of the objects within a particular subset of the index; conducting a scan of each of the subsets of the index to identify redundant replicas based on the scan; de-duplicating the redundant; and automatically copying an object from a device with a replica having an ongoing delete request to a device with a replica having been previously de-duplicated.
- a computer-readable memory may include computer-executable instructions.
- the computer-readable memory may include one or more instructions to conduct a scan of a portion of a index of objects in a distributed data replication system; one or more instructions to identify a redundant replica of one of the objects based on the scan of the portion of the index; one or more instructions to de-duplicate the redundant replica.
- Fig. 1 is a diagram of an exemplary network in which systems and methods described herein may be implemented
- Fig. 2 is a diagram of an exemplary configuration of the file system of Fig. 1;
- Fig. 3 is a diagram of exemplary components of a storage cluster of Fig. 1 ;
- Fig. 4 is a functional block diagram of an exemplary storage cluster of Fig. 1 ;
- Fig. 5 is a diagram of an exemplary record structure that may be used within an index of a distributed multi-master data replication system;
- FIGs. 6A-6B are flowcharts of exemplary processes for managing client-initiated upload/delete operations
- Fig. 7 is a flowchart of exemplary process for performing de-duplication in a distributed multi-master data replication system
- Fig. 8 is a flowchart of exemplary process for managing a delete request
- Fig. 9 is a flowchart of exemplary process for removing duplicate replicas
- Fig. 10 is a flowchart of exemplary process for optimizing bandwidth consumption and reducing latency in a distributed multi-master data replication system
- Fig. 1 1 is a diagram illustrating a portion of an exemplary global index according to an implementation described herein.
- Systems and/or methods described herein may provide an asynchronous distributed de- duplication algorithm for replicated storage clusters that provides availability, liveness and consistency guarantees for immutable objects.
- Implementations described herein may use the underlying replication layer of a distributed multi-master data replication system to replicate a content addressable index (also referred to herein as a "global index") between different storage clusters.
- Each object of the global index may have a unique content handle (e.g., a hash value or digital signature).
- the removal process of redundant replicas may keep at least one replica alive.
- Fig. 1 is a diagram of an exemplary system 100 in which systems and methods described herein may be implemented.
- System 100 may include clients 110-1 through 110-N (referred to collectively as clients 1 10, and individually as client 110) and storage clusters 120-1 through 120-M (referred to collectively as storage clusters 120, and individually as storage cluster 120) connected via a network 130.
- Storage clusters 120 may form a file system 140 (as shown by the dotted line in Fig.
- Network 130 may include one or more networks, such as a local area network (LAN), a wide area network (WAN), a telephone network (e.g., the Public Switched Telephone Network
- LAN local area network
- WAN wide area network
- telephone network e.g., the Public Switched Telephone Network
- PSTN public switched telephone network
- Clients 110 and storage clusters 120 may connect to network 130 via wired and/or wireless connections.
- Clients 1 10 may include one or more types of devices, such as a personal computer, a wireless telephone, a personal digital assistant (PDA), a lap top, or another type of communication device, and/or a thread or process running on one of these devices.
- a client 1 10 includes, or is linked to, an application on whose behalf client 110 communicates with storage cluster 120 to read or modify (e.g., write) file data.
- Storage cluster 120 may include one or more server devices, or other types of computation or communication devices, that may store, process, search, and/or provide information in a manner described herein.
- storage cluster 120 may include one or more servers (e.g., computer systems and/or applications) capable of maintaining a large-scale, random read/write-access data store for files.
- the data store of storage cluster 120 may permit an indexing system to quickly update portions of an index if a change occurs.
- the data store of storage cluster 120 may include one or more tables (e.g., a document table that may include one row per uniform resource locator (URL), auxiliary tables keyed by values other than URLs, etc.).
- URL uniform resource locator
- storage cluster 120 may be included in a distributed storage system (e.g., a "Bigtable” as set forth in Chang et al., “Bigtable: A Distributed Storage System for Structured Data," Proc. of the 7th OSDI, pp. 205-218 (Nov. 2006)) for managing structured data (e.g., a random-access storage cluster of documents) that may be designed to scale to a very large size (e.g., petabytes of data across thousands of servers).
- a distributed storage system e.g., a "Bigtable” as set forth in Chang et al., “Bigtable: A Distributed Storage System for Structured Data," Proc. of the 7th OSDI, pp. 205-218 (Nov. 2006)
- structured data e.g., a random-access storage cluster of documents
- a very large size e.g., petabytes of data across thousands of servers.
- system 100 may include a variety of other components, such as one or more dedicated consumer servers or hubs.
- a consumer server for example, may store a read-only copy of a data store from one or more storage clusters 120 for access by clients 110.
- a hub for example, may store a read-only copy of a data store from one or more storage clusters 120 for distribution to one or more consumer servers.
- EXEMPLARY STORAGE CLUSTER CONFIGURATION Fig.2 is a diagram of an exemplary configuration of the file system 140. As shown in
- file system 140 may include storage clusters 120-1, 120-2, 120-3, and 120-4.
- file system 140 may be a distributed multi-master data replication system, where each of storage clusters 120-1, 120-2, 120-3, and 120-4 may act as a master server for the other storage clusters.
- data may be replicated across storage clusters 120-1, 120-2, 120- 3, and 120-4 (e.g., in multiple geographical locations) to increase data availability and reduce network distance from clients (e.g., clients 110).
- distributed objects and references may be dynamically created, mutated, cloned and deleted in different storage clusters 120 and an underlying data replication layer (not shown) maintains the write-order fidelity to ensure that all storage clusters 120 will end up with the same version of data.
- the data replication layer respects the order of writes to the same replica for a single object.
- a global index of all of the objects in the distributed multi-master data replication system may be associated with each storage cluster 120.
- Each stored object may be listed by a unique content handle (such as a hash value, digital signature, etc.) in the global index.
- Selected storage clusters may each be assigned to be responsible for a distinct range of the content handles in the global index. For example, a single storage cluster 120 may be responsible for de-duplication of objects associated with particular content handles. Changes to the global index made by one storage cluster may be replicated to other storage clusters.
- file system 140 may contain fewer, additional, different, or differently arranged components than depicted in Fig. 2.
- one or more components of file system 140 may perform one or more tasks described as being performed by one or more other components of file system 140.
- FIG. 3 is a diagram of exemplary components of storage cluster 120.
- Storage cluster 120 may include a bus 310, a processor 320, a main memory 330, a read-only memory (ROM) 340, a storage device 350, an input device 360, an output device 370, and a communication interface 380.
- Bus 310 may include one or more conductors that permit communication among the components of storage cluster 120.
- Processor 320 may include any type of processor or microprocessor that may interpret and execute instructions.
- Main memory 330 may include a random access memory (RAM) or another type of dynamic storage device that may store information and instructions for execution by processor 320.
- ROM 340 may include a ROM device or another type of static storage device that may store static information and instructions for use by processor 320.
- Storage device 350 may include a magnetic and/or optical recording medium and its corresponding drive.
- storage device 350 may include one or more local disks 355 that provide persistent storage.
- storage cluster 120 may maintain metadata, for objects stored in file system 140, within one or more computer-readable mediums, such as main memory 330 and/or storage device 350.
- storage cluster 120 may store a global index within storage device 350 for all the objects stored within a distributed multi-master data replication system.
- Input device 360 may include one or more mechanisms that permit an operator to input information to storage cluster 120, such as a keyboard, a keypad, a button, a mouse, a pen, etc.
- Output device 370 may include one or more mechanisms that output information to the operator, including a display, a light emitting diode (LED), etc.
- Communication interface 380 may include any transceiver-like mechanism that enables storage cluster 120 to communicate with other devices and/or systems. For example, communication interface 380 may include mechanisms for communicating with other storage clusters 120 and/or clients 110.
- Fig. 4 illustrates a functional block diagram of storage cluster 120.
- storage cluster 120 may include data store 410 and de-duplication logic 420.
- data store 410 may be provided within storage cluster 120.
- some or all of data store 410 may be stored within one or more other devices of system 100 in communication with storage cluster 120, such as external memory devices or devices associated with an indexing system (not shown).
- Data store 410 may include a replicated index store 412 and a local object store 414.
- Replicated index store 412 may be included as part of the replication layer of the distributed multi- master data replication system.
- Replicated index store 412 may store information associated with the global index. At least a portion of replicated index store 412 may be replicated on multiple storage clusters 120. The number of replicas for each replicated index store 412 may be user-configurable.
- Local object store 414 may store objects locally within storage cluster 120.
- Local object store 414 may include files, such as images or videos uploaded by clients (e.g., clients 110).
- De-duplication logic 420 may include logic to remove redundant replicas from storage clusters within the distributed multi-master data replication system (e.g., storage clusters 120-1, 120- 2, 120-3, and 120-4). De-duplication logic 420 for each participating storage cluster may be assigned to be responsible for a particular section of the global index. For example, de-duplication logic 420 may be assigned to a particular range of content handles for the global index. Thus, only one storage cluster within the distributed multi-master data replication system may be able to perform destructive operations (e.g., deletion of replicas) on a replicated object within the system. To facilitate de-duplication, records may be generated by de-duplication logic 420 and appended to a portion of the global index associated with a particular content handle.
- Records may include, for example, a "Data” designator for initiating a live replica, a "DeleteRequest” designator for indicating an ongoing delete request for a replica, and a "Deduped” designator for indicating a replica that has been selected for de-duplication. Record formats and uses are described in more detail below.
- Fig. 4 shows exemplary functional components of storage cluster 120
- storage cluster 120 may contain fewer, additional, different, or differently arranged functional components than depicted in Fig. 4.
- one or more functional components of storage cluster 120 may perform one or more other tasks described as being performed by one or more other functional components.
- Fig. 5 provides an illustration of an exemplary record structure 500 for a de-duplication designation record that may be written to the global index in an exemplary implementation.
- the de- duplication designation record may be associated in the global index with a particular content handle of an object replica.
- record structure 500 may include storage cluster identifier ("ID") section 510, a storage location section 520, and designation section 530.
- Storage cluster identification section 510 may include a unique identification (e.g., "Cluster ID") for the storage cluster 120 that is storing the object replica for which the record is being written.
- Location section 520 may include an address for the location of the replica within storage cluster 120 that is identified by storage cluster identification section 510.
- Designation section 530 may include, for example, a "Data” designator, a "DeleteRequest” designator, or a "Deduped” designator.
- Record structure 500 may be listed in the form of "ClusterlDiLocationiDesignation.”
- a record for a replica may be added to the global index by storage cluster 120-1 with the record "01 :234523/2000:DeleteRequest,” where "01” is the cluster ID for storage cluster 120-1 , “234523/2000” is the location, within storage cluster 120-1 at which the replica is stored, and
- a record for another replica of the same object in storage cluster 120-2 may be "02:234544/1000:Data,” where "02” is the cluster ID for storage cluster 120-2, "234544/1000” is the location within storage cluster 120-2, and "Data” is the designator.
- Figs. 6A and 6B are flowcharts of exemplary processes for managing client-initiated upload/delete operations.
- Fig. 6A depicts a flowchart for an exemplary process 600 of uploading an object from a client.
- Fig. 6B depicts a flowchart for an exemplary process 650 of removing an object deleted by a client.
- processes 600 and 650 may be performed by one of storage clusters 120.
- Processes 600 and 650 may be implemented in response to client (e.g., client 1 10) activities.
- client e.g., client 1
- process 600 may begin when an uploaded file is received from a client (block 610).
- storage cluster 120-1 may receive a new file from one of clients 1 10.
- the uploaded file may be stored (block 620) and a "Data" designator for the uploaded file may be written to the global index (block 630).
- storage cluster 120-1 may store the uploaded file in a memory (e.g., storage device 350) and add a content handle for the object to the global index.
- Storage cluster 120-1 may also write a data record (e.g., "01 :Location:Data”) to the replicated global index addressed by the content handle of the object.
- a data record e.g., "01 :Location:Data
- process 650 may begin when a notice of a deleted file is received (block 660).
- storage cluster 120-1 may receive an indication that one of clients 110 has deleted a file.
- a delete request may be initiated (block 670) and a "DeleteRequest" designator for the deleted file may be written to the global index (block 680).
- storage cluster 120-1 may initiate a delete request to asynchronously remove the delete file from file system 140.
- Storage device 120-1 may also write a "DeleteRequest" record (e.g., "01 :Location:DeleteReqeust”) to the replicated global index addressed by the content handle of the object.
- Fig. 7 is a flowchart of an exemplary process 700 for performing de-duplication in a distributed multi-master data replication system (e.g., file system 140).
- process 700 may be performed by one of storage clusters 120.
- some or all of process 700 may be performed by another device or a group of devices, including or excluding storage cluster 120.
- Process 700 may be implemented periodically in each storage cluster 120 and may include a scan of all or a portion of the objects in the storage cluster 120.
- storage clusters 120-1 and 120-2 of file system 140 where storage cluster 120-1 includes a cluster ID of "01" and storage cluster 120-2 includes a cluster ID of "02."
- process 700 may begin with conducting a scan of the global index
- storage cluster 120-1 may conduct a scan of all or a portion of the objects listed in the global index.
- the scan may identify, for example, multiple replicas and/or objects marked for deletion.
- a delete request It may be determined if a delete request is encountered (block 720). For example, storage cluster 120-1 may encounter an object in the global index that includes a delete request designator (e.g., "02:Location:DeleteReqeust") from another storage cluster (e.g., from storage cluster 120-2). If it is determined that a delete request is encountered (block 720-YES), then the delete request may be processed (block 730). For example, storage cluster 120-1 may process the delete request as described in more detail with respect to Fig. 8.
- a delete request designator e.g., "02:Location:DeleteReqeust”
- Redundant replicas may be replicated objects in different locations that have no outstanding delete requests for the object.
- storage cluster 120-1 may identify multiple replicas for the same object that correspond to a content handle for which storage cluster 120-1 is responsible.
- the multiple replicas may be stored, for example, in different storage clusters (e.g., storage cluster 120-1 and storage cluster 120-2) or in different locations within the same storage cluster.
- the redundant replicas(s) may be removed (block 750).
- storage cluster 120-1 may remove the redundant replica(s) as described in more detail with respect to Fig. 9. If it is determined that redundant replicas do not exist (block 740-NO), then the process may return to block 710, where another scan of the global index may be conducted (block 710).
- Fig. 8 illustrates exemplary operations associated with the processing of a delete request of block 660 of Fig. 6.
- a delete request may be encountered for an object (block 810).
- a scan being conducted by storage cluster 120-1 may identify a content handle in the global index with a delete request designator previously written by storage cluster 120-1 to delete a replica in a certain storage cluster (e.g., "02:Location:DeleteRequest").
- storage cluster 120-1 may apply operations to determine if the replica can now be de-duplicated.
- a de-duplication designator exists (block 820). For example, storage cluster 120-1 may review other records in the global index associated with the content handle to determine if a de-duplication designator exists (e.g., 02:Location:Deduped")- If it is determined that a de-duplication designator exists (block 820 - YES), then the replica and the related records in the global index may be de-duplicated (block 830). For example, storage cluster 120-1 may initiate a delete request to delete the replica in storage cluster 120-2 (if any) and delete any records (e.g., "02:Location:*", where "*" may be any designator) from the global index that relate to the content handle for the deleted replica.
- a de-duplication designator e.g. 02:Location:Deduped
- storage cluster 120-1 may review the content handle for the global index to determine whether another live replica exists for the object.
- the global index may include, for example, a data record for that content handle from another storage cluster (e.g., "03:Location:Data").
- the replica may be de-duplicated as described above with respect to block 830. If another live replica does not exist (block 840 - NO), then it may be determined if all replicas have delete requests (block 850). For example, storage cluster 120-1 may review the content handle for the global index to determine whether all the replicas associated with the content handle have an outstanding delete request (e.g., "*:*:DeleteRequest", where "*" may be any ClusterID and any location, respectively).
- the replica may be de-duplicated as described above with respect to block 830. If it is determined that all replicas do not have delete requests (block 850 -NO), then the object may be copied from a storage cluster that initiated a delete request to a different storage cluster and the global index may be updated (block 860). For example, in response to the record "02:Location:DeleteRequest," storage cluster 120-1 may copy the object from storage cluster 120-2 to another storage cluster 120-3 for which there is a de-duplication record (e.g., "03:Location:Deduped") and no outstanding delete request. Storage cluster 120-1 may delete the previous de-duplication record (e.g., "03: Location :Deduped”) associated with the replica and write a data designator (e.g.,
- Fig. 9 illustrates exemplary operations associated with the removing of duplicate references of block 750 of Fig. 7.
- Multiple replicas with no delete requests may be identified (block 910).
- storage cluster 120-1 may review the global index and identify two or more replicas that have no outstanding delete requests corresponding to a content handle for which storage cluster 120-01 is responsible.
- Criteria to determine replica(s) to be de-duplicated may be applied (block 920).
- storage cluster 120-1 may apply criteria to de-duplicate the redundant replica that may be stored within storage cluster 120-1.
- the criteria to de-duplicate redundant replicas may be based on a variety of factors, such as geographic proximity of the replicas, available storage capacity at a storage cluster, or other factors.
- Storage cluster 120-1 (e.g., using de-duplication logic 420) may apply the criteria to the two or more replicas that have no outstanding delete requests identified above.
- multiple replicas may be identified to be de-duplicated.
- storage cluster 120-1 may leave more than one live replica (e.g., a replica not marked for de- duplication).
- the global index may be updated to designate de-duplicated replica(s) as "Deduped” (block 930). For example, for each de-duplicated replica, storage cluster 120-1 may delete the previous data record (e.g., "02:Location:Data") associated with the replica and write a de-duplication designator (e.g., "02:Location:Deduped") to the corresponding content handle in the global index. De-duplication of the redundant replicas may be accomplished using de-duplication messages that are replicated as a part of the global index. The replicas marked for de-duplication may be stored within storage cluster 120-1 or within another storage cluster (e.g., storage cluster 120- 2, 120-3, 120-4, etc.).
- storage cluster 120-1 may delete locally-stored replicas and the corresponding "01 :Location:Data" record from the global index and add "01 :Location:Deduped" to the global index.
- Storage cluster 120-1 may also initiate delete messages, using the replicated global index, to delete replicas stored in other clusters.
- Fig. 10 provides a flowchart of an exemplary process 1000 for optimizing bandwidth consumption and reducing latency in a distributed multi-master data replication system (e.g., file system 140).
- process 1000 may be performed by one of storage clusters 120.
- some or all of process 1000 may be performed by another device or group of devices, including or excluding storage cluster 120.
- process 1000 may begin with receiving a request for an object (block 1010).
- storage cluster 120-1 may receive a request from a client (e.g., client 110-1) to obtain an object.
- Object locations may be looked up in the global index (block 1020).
- storage cluster 120-1 may look up the replica location(s) for the object in the replicated global index using the content handle of the object.
- the "best" replica location may be identified (block 1030). For example, assuming that more than one replica is available, storage cluster 120-1 may determine the "best" replica to retrieve to minimize network resources.
- the "best" replica may be the replica that has the closest geographic location to storage cluster 120-1. In other implementations, the "best” replica may be based on a combination of available network connectivity, geographic location, and/or other criteria.
- the "best" replica for the object may be stored locally within storage cluster 120-1.
- the object may be retrieved from the identified location (block 1040).
- storage cluster 120-1 may request the "best" replica from the closest available storage cluster and receive the replica to satisfy the client request. Storage cluster 120-1 may then send the replica to the client.
- Fig. 11 provides a portion 1100 of an exemplary global index according to an implementation described herein.
- the index may include, among other information, a content handle column 1110 and a De-duplication designation record column 1120.
- a distributed multi-master data replication system includes three storage clusters, XX, YY, and ZZ.
- a de-duplication algorithm may run periodically in each of storage clusters XX, YY, and ZZ and may scan all or a portion of the global index.
- records e.g., Data, DeleteRequest, and Deduped
- index portion 1100 includes content handles and associated delete designation records for four objects.
- "Handlel 1” has records indicating replicas are stored at storage cluster XX ("XX:Location01 :Data”) and storage cluster YY ("YY:LocationO 1 :Data”), respectively.
- "Handle21” has a record indicating a replica is stored at storage cluster XX (“XX:Location02:Data”) and another replica at storage cluster YY has an ongoing delete request
- “Handle41” has records indicating a replica is stored at storage cluster YY ("XX:Location04:Data”) and a record indicating the replica with an ongoing delete request at storage cluster YY ("YY:Location04:DeleteRequest”). Handle41 also has one record indicating de-duplication of a replica has occurred (“ZZ:Location02:Deduped”).
- the de-duplication algorithm used by the storage clusters can operate using guidelines consistent with the principles described herein. Assume storage cluster XX is assigned responsibility for the portion of the global index including "Handlel 1,” ⁇ andle21,” “Handle31,” and “Handle41.”
- the storage cluster may write a data record (e.g., "ClusterID:Location:Data”) to the replicated global index addressed by the content handle of the object.
- a data record e.g., "ClusterID:Location:Data”
- “XX:Location01:Data” and “YY:Location01 :Data” illustrate data records for replicas of "Handlel 1.”
- "XX:Location02:Data” illustrates a data record for a replica of "Handle21.” Similar data records can be seen for "Handle31" and "Handle 41.”
- the storage cluster may look up the replica locations in the replicated global index using the content handle of the object and fetch the replica from the "best” (e.g., closest) cluster. For example, assuming an object corresponding to "Handlel l" is requested at storage cluster ZZ and that storage cluster YY is closer to storage cluster ZZ than is storage cluster XX, storage cluster ZZ may request the object replica corresponding to "Handle 11" from storage cluster YY.
- the storage cluster may write "ClusterID:Location:DeleteRequest” to the replicated global index addressed by the content handle of the object.
- ClusterID:Location:DeleteRequest For example, "YY:Location02:DeleteRequest” illustrates a record for a deleted replica of "Handle21" in storage cluster YY.
- YY:Location03:DeleteRequest and “ZZ:Location:01 :DeleteRequest” illustrate records for deleted replicas of "Handle31" for storage clusters YY and ZZ, respectively.
- the storage cluster may delete redundant replicas of the object (possibly leaving more than one live replica). For each deleted replica in another storage cluster, the storage cluster may delete the data record and write a de-duplication record. For example, the scan in storage cluster XX may identify that "Handlel 1" has records indicating replicas are stored at storage cluster XX ("XX:Location01 :Data") and storage cluster YY ("YY:Location01 :Data”), respectively. Based on criteria provided for removing redundant references, storage cluster XX may initiate deletion of the replica at storage cluster YY. Storage cluster XX may delete the record "YY:Location01:Data" shown in Fig. 1 1 and write "YY:Location01 :Deduped" instead.
- storage cluster XX may apply the following analysis.
- a delete request e.g., "ClusterID:Location:DeleteRequest”
- a replica in another storage cluster e.g., storage cluster YY or ZZ
- storage cluster XX may apply the following analysis.
- the storage cluster XX can delete the replica of the object in storage cluster YY or ZZ (if any) and delete the records "YY:Location:*" or "ZZrLocation:*.”
- the replica for "Handle21" in storage cluster YY and the record "YY:Location02:DeleteRequest” may be deleted by storage cluster XX since another live object (indicated by the record "XX:Location02:Data”) exists.
- replica for "Handle31" in storage cluster YY and the record “YY:Location:03:DeleteRequest” may be deleted by storage cluster XX since both replicas in storage cluster YY and storage cluster ZZ have outstanding delete requests.
- storage cluster XX can copy the object from YY or ZZ to another storage cluster for which there is a de-duplication record and no outstanding delete request, deleting the de-duplication record and writing a data record.
- the replica for "Handle41 " in storage cluster YY (“YY:Location04:DeleteRequest”) may trigger storage cluster XX to copy the object associated with "Handle41" to storage cluster ZZ.
- Storage cluster XX may update the global index to change "ZZ:Location02:Deduped" to "ZZ:Location02:Data.”
- the correctness of the algorithm is straightforward as all deletion operations on the object are performed only by the scan process in the storage cluster responsible for its content handle.
- the algorithm also transparently deals with multiple object replicas in the same cluster that have different locations (e.g. XX:Locationl and XX:Location2).
- Systems and/or methods described herein may store a global index of objects in a distributed data replication system and replicate the global index and some of the objects throughout the distributed data replication system.
- a storage cluster may be assigned as the responsible entity for de-duplication within a particular subset of the global index.
- the storage cluster may conduct a scan of the subset of the global index and identify redundant replicas based on the scan.
- the storage cluster may de-duplicate the redundant replicas stored locally or in a remote storage cluster.
- a synchronous version of the de-duplication algorithm may be used in which different storage clusters communicate directly rather than using the replication layer within a distributed data replication system.
- This logic or component may include hardware, such as a processor, microprocessor, an application specific integrated circuit or a field programmable gate array, or a combination of hardware and software (e.g., software executed by a processor). It should be emphasized that the term “comprises” and/or “comprising” when used in this specification is taken to specify the presence of stated features, integers, steps, or components, but does not preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2009330073A AU2009330073B2 (en) | 2008-12-22 | 2009-12-22 | Asynchronous distributed de-duplication for replicated content addressable storage clusters |
BRPI0922990-6A BRPI0922990B1 (pt) | 2008-12-22 | 2009-12-22 | Sistema e métodos de armazenamento, varredura e desduplicação de índices de objetos em sistema de replicação de dados distribuídos, dispositivo de pluralidade de dispositivos em sistema de replicação de dados distribuídos e memória legível em computador |
EP09799831.4A EP2368199B1 (en) | 2008-12-22 | 2009-12-22 | Asynchronous distributed de-duplication for replicated content addressable storage clusters |
JP2011542576A JP5579195B2 (ja) | 2008-12-22 | 2009-12-22 | 複製されたコンテンツアドレス可能ストレージクラスタのための非同期分散型重複排除 |
CA2747746A CA2747746C (en) | 2008-12-22 | 2009-12-22 | Asynchronous distributed de-duplication for replicated content addressable storage clusters |
CN200980156970.8A CN102317938B (zh) | 2008-12-22 | 2009-12-22 | 用于复制内容可寻址存储集群的异步分布式去重 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13985708P | 2008-12-22 | 2008-12-22 | |
US61/139,857 | 2008-12-22 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2010075407A1 true WO2010075407A1 (en) | 2010-07-01 |
Family
ID=41719028
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2009/069234 WO2010075407A1 (en) | 2008-12-22 | 2009-12-22 | Asynchronous distributed de-duplication for replicated content addressable storage clusters |
Country Status (9)
Country | Link |
---|---|
US (5) | US8712974B2 (ja) |
EP (1) | EP2368199B1 (ja) |
JP (2) | JP5579195B2 (ja) |
CN (2) | CN104166673B (ja) |
AU (1) | AU2009330073B2 (ja) |
BR (1) | BRPI0922990B1 (ja) |
CA (1) | CA2747746C (ja) |
DE (1) | DE202009019139U1 (ja) |
WO (1) | WO2010075407A1 (ja) |
Families Citing this family (239)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8645137B2 (en) | 2000-03-16 | 2014-02-04 | Apple Inc. | Fast, language-independent method for user authentication by voice |
US8677377B2 (en) | 2005-09-08 | 2014-03-18 | Apple Inc. | Method and apparatus for building an intelligent automated assistant |
US8190742B2 (en) * | 2006-04-25 | 2012-05-29 | Hewlett-Packard Development Company, L.P. | Distributed differential store with non-distributed objects and compression-enhancing data-object routing |
US9318108B2 (en) | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
CA2705379C (en) * | 2006-12-04 | 2016-08-30 | Commvault Systems, Inc. | Systems and methods for creating copies of data, such as archive copies |
US7840537B2 (en) | 2006-12-22 | 2010-11-23 | Commvault Systems, Inc. | System and method for storing redundant information |
US8977255B2 (en) | 2007-04-03 | 2015-03-10 | Apple Inc. | Method and system for operating a multi-function portable electronic device using voice-activation |
US10002189B2 (en) | 2007-12-20 | 2018-06-19 | Apple Inc. | Method and apparatus for searching using an active ontology |
US9330720B2 (en) | 2008-01-03 | 2016-05-03 | Apple Inc. | Methods and apparatus for altering audio output signals |
US8996376B2 (en) | 2008-04-05 | 2015-03-31 | Apple Inc. | Intelligent text-to-speech conversion |
US8103628B2 (en) * | 2008-04-09 | 2012-01-24 | Harmonic Inc. | Directed placement of data in a redundant data storage system |
US10496753B2 (en) | 2010-01-18 | 2019-12-03 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US8219524B2 (en) * | 2008-06-24 | 2012-07-10 | Commvault Systems, Inc. | Application-aware and remote single instance data management |
US9098495B2 (en) | 2008-06-24 | 2015-08-04 | Commvault Systems, Inc. | Application-aware and remote single instance data management |
US8166263B2 (en) | 2008-07-03 | 2012-04-24 | Commvault Systems, Inc. | Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices |
US20100030549A1 (en) | 2008-07-31 | 2010-02-04 | Lee Michael M | Mobile device having human language translation capability with positional feedback |
WO2010036754A1 (en) | 2008-09-26 | 2010-04-01 | Commvault Systems, Inc. | Systems and methods for managing single instancing data |
US9015181B2 (en) | 2008-09-26 | 2015-04-21 | Commvault Systems, Inc. | Systems and methods for managing single instancing data |
US8676904B2 (en) | 2008-10-02 | 2014-03-18 | Apple Inc. | Electronic devices with voice command and contextual data processing capabilities |
US8412677B2 (en) * | 2008-11-26 | 2013-04-02 | Commvault Systems, Inc. | Systems and methods for byte-level or quasi byte-level single instancing |
WO2010067118A1 (en) | 2008-12-11 | 2010-06-17 | Novauris Technologies Limited | Speech recognition involving a mobile device |
JP5579195B2 (ja) * | 2008-12-22 | 2014-08-27 | グーグル インコーポレイテッド | 複製されたコンテンツアドレス可能ストレージクラスタのための非同期分散型重複排除 |
US8401996B2 (en) | 2009-03-30 | 2013-03-19 | Commvault Systems, Inc. | Storing a variable number of instances of data objects |
US9141621B2 (en) | 2009-04-30 | 2015-09-22 | Hewlett-Packard Development Company, L.P. | Copying a differential data store into temporary storage media in response to a request |
US20100281077A1 (en) * | 2009-04-30 | 2010-11-04 | Mark David Lillibridge | Batching requests for accessing differential data stores |
US8578120B2 (en) | 2009-05-22 | 2013-11-05 | Commvault Systems, Inc. | Block-level single instancing |
US10241644B2 (en) | 2011-06-03 | 2019-03-26 | Apple Inc. | Actionable reminder entries |
US10706373B2 (en) | 2011-06-03 | 2020-07-07 | Apple Inc. | Performing actions associated with task items that represent tasks to perform |
US10241752B2 (en) | 2011-09-30 | 2019-03-26 | Apple Inc. | Interface for a virtual digital assistant |
US9858925B2 (en) | 2009-06-05 | 2018-01-02 | Apple Inc. | Using context information to facilitate processing of commands in a virtual assistant |
US9431006B2 (en) | 2009-07-02 | 2016-08-30 | Apple Inc. | Methods and apparatuses for automatic speech recognition |
US8572137B2 (en) * | 2009-09-08 | 2013-10-29 | International Business Machines Corporation | Data de-duplication in a distributed network |
US10553209B2 (en) | 2010-01-18 | 2020-02-04 | Apple Inc. | Systems and methods for hands-free notification summaries |
US10705794B2 (en) | 2010-01-18 | 2020-07-07 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US10679605B2 (en) | 2010-01-18 | 2020-06-09 | Apple Inc. | Hands-free list-reading by intelligent automated assistant |
US8682667B2 (en) | 2010-02-25 | 2014-03-25 | Apple Inc. | User profiling for selecting user specific voice input processing information |
US9135016B1 (en) * | 2010-05-14 | 2015-09-15 | Emc Corporation | Methods and apparatus for managing replication of objects in a storage environment |
WO2012045023A2 (en) | 2010-09-30 | 2012-04-05 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US10762293B2 (en) | 2010-12-22 | 2020-09-01 | Apple Inc. | Using parts-of-speech tagging and named entity recognition for spelling correction |
US9262612B2 (en) | 2011-03-21 | 2016-02-16 | Apple Inc. | Device access using voice authentication |
US8433681B2 (en) | 2011-05-12 | 2013-04-30 | Dell Products L.P. | System and method for managing replication in an object storage system |
US10057736B2 (en) | 2011-06-03 | 2018-08-21 | Apple Inc. | Active transport based notifications |
US10672399B2 (en) | 2011-06-03 | 2020-06-02 | Apple Inc. | Switching between text data and audio data based on a mapping |
US8994660B2 (en) | 2011-08-29 | 2015-03-31 | Apple Inc. | Text correction processing |
US9286052B1 (en) * | 2011-09-15 | 2016-03-15 | Emc Corporation | Upgrading software on a pair of nodes in a clustered environment |
US20130073286A1 (en) * | 2011-09-20 | 2013-03-21 | Apple Inc. | Consolidating Speech Recognition Results |
US8468138B1 (en) | 2011-12-02 | 2013-06-18 | International Business Machines Corporation | Managing redundant immutable files using deduplication in storage clouds |
US8682868B2 (en) * | 2011-12-02 | 2014-03-25 | International Business Machines Corporation | Removal of data remanence in deduplicated storage clouds |
US20130151483A1 (en) * | 2011-12-07 | 2013-06-13 | Quantum Corporation | Adaptive experience based De-duplication |
US8504595B2 (en) * | 2011-12-07 | 2013-08-06 | Jeffrey Tofano | De-duplication for a global coherent de-duplication repository |
KR101583748B1 (ko) * | 2011-12-08 | 2016-01-19 | 엠파이어 테크놀로지 디벨롭먼트 엘엘씨 | 사용자 간의 중복제거를 허용하기 위한 저장소 할인 |
US10134385B2 (en) | 2012-03-02 | 2018-11-20 | Apple Inc. | Systems and methods for name pronunciation |
US9483461B2 (en) | 2012-03-06 | 2016-11-01 | Apple Inc. | Handling speech synthesis of content for multiple languages |
US8732124B1 (en) * | 2012-03-26 | 2014-05-20 | Emc Corporation | Multisite replication with coordinated cycle switching |
US9020890B2 (en) | 2012-03-30 | 2015-04-28 | Commvault Systems, Inc. | Smart archiving and data previewing for mobile devices |
US9280610B2 (en) | 2012-05-14 | 2016-03-08 | Apple Inc. | Crowd sourcing information to fulfill user requests |
US10417037B2 (en) | 2012-05-15 | 2019-09-17 | Apple Inc. | Systems and methods for integrating third party services with a digital assistant |
US8984028B2 (en) | 2012-05-29 | 2015-03-17 | Recommind, Inc. | Systems and methods for storing data and eliminating redundancy |
US9721563B2 (en) | 2012-06-08 | 2017-08-01 | Apple Inc. | Name recognition system |
CN103491124B (zh) * | 2012-06-14 | 2018-08-14 | 南京中兴软件有限责任公司 | 一种对彩信数据进行处理的方法及分布式缓存系统 |
US9495129B2 (en) | 2012-06-29 | 2016-11-15 | Apple Inc. | Device, method, and user interface for voice-activated navigation and browsing of a document |
US9086819B2 (en) * | 2012-07-25 | 2015-07-21 | Anoosmar Technologies Private Limited | System and method for combining deduplication and encryption of data |
US20140067775A1 (en) * | 2012-09-05 | 2014-03-06 | salesforce.com,inc | System, method and computer program product for conditionally performing de-duping on data |
US9576574B2 (en) | 2012-09-10 | 2017-02-21 | Apple Inc. | Context-sensitive handling of interruptions by intelligent digital assistant |
US9547647B2 (en) | 2012-09-19 | 2017-01-17 | Apple Inc. | Voice-based media searching |
US9633022B2 (en) | 2012-12-28 | 2017-04-25 | Commvault Systems, Inc. | Backup and restoration for a deduplicated file system |
KR20240132105A (ko) | 2013-02-07 | 2024-09-02 | 애플 인크. | 디지털 어시스턴트를 위한 음성 트리거 |
US9368114B2 (en) | 2013-03-14 | 2016-06-14 | Apple Inc. | Context-sensitive handling of interruptions |
US10652394B2 (en) | 2013-03-14 | 2020-05-12 | Apple Inc. | System and method for processing voicemail |
WO2014144579A1 (en) | 2013-03-15 | 2014-09-18 | Apple Inc. | System and method for updating an adaptive speech recognition model |
US10748529B1 (en) | 2013-03-15 | 2020-08-18 | Apple Inc. | Voice activated device for use with a voice-based digital assistant |
AU2014233517B2 (en) | 2013-03-15 | 2017-05-25 | Apple Inc. | Training an at least partial voice command system |
WO2014197336A1 (en) | 2013-06-07 | 2014-12-11 | Apple Inc. | System and method for detecting errors in interactions with a voice-based digital assistant |
US9582608B2 (en) | 2013-06-07 | 2017-02-28 | Apple Inc. | Unified ranking with entropy-weighted information for phrase-based semantic auto-completion |
WO2014197334A2 (en) | 2013-06-07 | 2014-12-11 | Apple Inc. | System and method for user-specified pronunciation of words for speech synthesis and recognition |
WO2014197335A1 (en) | 2013-06-08 | 2014-12-11 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US10176167B2 (en) | 2013-06-09 | 2019-01-08 | Apple Inc. | System and method for inferring user intent from speech inputs |
KR101772152B1 (ko) | 2013-06-09 | 2017-08-28 | 애플 인크. | 디지털 어시스턴트의 둘 이상의 인스턴스들에 걸친 대화 지속성을 가능하게 하기 위한 디바이스, 방법 및 그래픽 사용자 인터페이스 |
EP3008964B1 (en) | 2013-06-13 | 2019-09-25 | Apple Inc. | System and method for emergency calls initiated by voice command |
EP3028157A1 (en) * | 2013-07-29 | 2016-06-08 | Hewlett Packard Enterprise Development LP | Integrity of frequently used de-duplication objects |
DE112014003653B4 (de) | 2013-08-06 | 2024-04-18 | Apple Inc. | Automatisch aktivierende intelligente Antworten auf der Grundlage von Aktivitäten von entfernt angeordneten Vorrichtungen |
US10296160B2 (en) | 2013-12-06 | 2019-05-21 | Apple Inc. | Method for extracting salient dialog usage from live data |
US10324897B2 (en) | 2014-01-27 | 2019-06-18 | Commvault Systems, Inc. | Techniques for serving archived electronic mail |
US9430490B1 (en) * | 2014-03-28 | 2016-08-30 | Formation Data Systems, Inc. | Multi-tenant secure data deduplication using data association tables |
US9620105B2 (en) | 2014-05-15 | 2017-04-11 | Apple Inc. | Analyzing audio input for efficient speech and music recognition |
US10592095B2 (en) | 2014-05-23 | 2020-03-17 | Apple Inc. | Instantaneous speaking of content on touch devices |
US9502031B2 (en) | 2014-05-27 | 2016-11-22 | Apple Inc. | Method for supporting dynamic grammars in WFST-based ASR |
US9430463B2 (en) | 2014-05-30 | 2016-08-30 | Apple Inc. | Exemplar-based natural language processing |
US9633004B2 (en) | 2014-05-30 | 2017-04-25 | Apple Inc. | Better resolution when referencing to concepts |
US9734193B2 (en) | 2014-05-30 | 2017-08-15 | Apple Inc. | Determining domain salience ranking from ambiguous words in natural speech |
US10289433B2 (en) | 2014-05-30 | 2019-05-14 | Apple Inc. | Domain specific language for encoding assistant dialog |
US10170123B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Intelligent assistant for home automation |
CN110797019B (zh) | 2014-05-30 | 2023-08-29 | 苹果公司 | 多命令单一话语输入方法 |
US10078631B2 (en) | 2014-05-30 | 2018-09-18 | Apple Inc. | Entropy-guided text prediction using combined word and character n-gram language models |
US9715875B2 (en) | 2014-05-30 | 2017-07-25 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US9760559B2 (en) | 2014-05-30 | 2017-09-12 | Apple Inc. | Predictive text input |
US9842101B2 (en) | 2014-05-30 | 2017-12-12 | Apple Inc. | Predictive conversion of language input |
US9785630B2 (en) | 2014-05-30 | 2017-10-10 | Apple Inc. | Text prediction using combined word N-gram and unigram language models |
US9338493B2 (en) | 2014-06-30 | 2016-05-10 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US10659851B2 (en) | 2014-06-30 | 2020-05-19 | Apple Inc. | Real-time digital assistant knowledge updates |
US10446141B2 (en) | 2014-08-28 | 2019-10-15 | Apple Inc. | Automatic speech recognition based on user feedback |
GB2529859A (en) | 2014-09-04 | 2016-03-09 | Ibm | Device and method for storing data in a distributed file system |
US9818400B2 (en) | 2014-09-11 | 2017-11-14 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US10789041B2 (en) | 2014-09-12 | 2020-09-29 | Apple Inc. | Dynamic thresholds for always listening speech trigger |
US11095715B2 (en) | 2014-09-24 | 2021-08-17 | Ebay Inc. | Assigning storage responsibility in a distributed data storage system with replication |
US9800575B1 (en) * | 2014-09-24 | 2017-10-24 | Ebay Inc. | Assigning storage responsibility in a distributed data storage system with replication |
US9646609B2 (en) | 2014-09-30 | 2017-05-09 | Apple Inc. | Caching apparatus for serving phonetic pronunciations |
US9668121B2 (en) | 2014-09-30 | 2017-05-30 | Apple Inc. | Social reminders |
US10074360B2 (en) | 2014-09-30 | 2018-09-11 | Apple Inc. | Providing an indication of the suitability of speech recognition |
US9886432B2 (en) | 2014-09-30 | 2018-02-06 | Apple Inc. | Parsimonious handling of word inflection via categorical stem + suffix N-gram language models |
US10127911B2 (en) | 2014-09-30 | 2018-11-13 | Apple Inc. | Speaker identification and unsupervised speaker adaptation techniques |
US10552013B2 (en) | 2014-12-02 | 2020-02-04 | Apple Inc. | Data detection |
US9711141B2 (en) | 2014-12-09 | 2017-07-18 | Apple Inc. | Disambiguating heteronyms in speech synthesis |
CN104462396B (zh) * | 2014-12-10 | 2017-12-19 | 北京国双科技有限公司 | 字符串处理方法和装置 |
US10152299B2 (en) | 2015-03-06 | 2018-12-11 | Apple Inc. | Reducing response latency of intelligent automated assistants |
US9865280B2 (en) | 2015-03-06 | 2018-01-09 | Apple Inc. | Structured dictation using intelligent automated assistants |
US9886953B2 (en) | 2015-03-08 | 2018-02-06 | Apple Inc. | Virtual assistant activation |
US9721566B2 (en) | 2015-03-08 | 2017-08-01 | Apple Inc. | Competing devices responding to voice triggers |
US10567477B2 (en) | 2015-03-08 | 2020-02-18 | Apple Inc. | Virtual assistant continuity |
US9899019B2 (en) | 2015-03-18 | 2018-02-20 | Apple Inc. | Systems and methods for structured stem and suffix language models |
US9842105B2 (en) | 2015-04-16 | 2017-12-12 | Apple Inc. | Parsimonious continuous-space phrase representations for natural language processing |
US10460227B2 (en) | 2015-05-15 | 2019-10-29 | Apple Inc. | Virtual assistant in a communication session |
US10324914B2 (en) | 2015-05-20 | 2019-06-18 | Commvalut Systems, Inc. | Handling user queries against production and archive storage systems, such as for enterprise customers having large and/or numerous files |
US10083688B2 (en) | 2015-05-27 | 2018-09-25 | Apple Inc. | Device voice control for selecting a displayed affordance |
US10127220B2 (en) | 2015-06-04 | 2018-11-13 | Apple Inc. | Language identification from short strings |
US10101822B2 (en) | 2015-06-05 | 2018-10-16 | Apple Inc. | Language input correction |
US9578173B2 (en) | 2015-06-05 | 2017-02-21 | Apple Inc. | Virtual assistant aided communication with 3rd party service in a communication session |
US11025565B2 (en) | 2015-06-07 | 2021-06-01 | Apple Inc. | Personalized prediction of responses for instant messaging |
US10255907B2 (en) | 2015-06-07 | 2019-04-09 | Apple Inc. | Automatic accent detection using acoustic models |
US10186254B2 (en) | 2015-06-07 | 2019-01-22 | Apple Inc. | Context-based endpoint detection |
US20160378747A1 (en) | 2015-06-29 | 2016-12-29 | Apple Inc. | Virtual assistant for media playback |
CN105187517A (zh) * | 2015-08-20 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | 一种实现元数据集群的方法及元数据集群 |
US10671428B2 (en) | 2015-09-08 | 2020-06-02 | Apple Inc. | Distributed personal assistant |
US10747498B2 (en) | 2015-09-08 | 2020-08-18 | Apple Inc. | Zero latency digital assistant |
US10089358B2 (en) * | 2015-09-21 | 2018-10-02 | The Nielsen Company (Us), Llc | Methods and apparatus to partition data |
US9697820B2 (en) | 2015-09-24 | 2017-07-04 | Apple Inc. | Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks |
US10366158B2 (en) | 2015-09-29 | 2019-07-30 | Apple Inc. | Efficient word encoding for recurrent neural network language models |
US11010550B2 (en) | 2015-09-29 | 2021-05-18 | Apple Inc. | Unified language modeling framework for word prediction, auto-completion and auto-correction |
US11587559B2 (en) | 2015-09-30 | 2023-02-21 | Apple Inc. | Intelligent device identification |
US10042885B2 (en) * | 2015-10-29 | 2018-08-07 | International Business Machines Corporation | Index table based routing for query resource optimization |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10049668B2 (en) | 2015-12-02 | 2018-08-14 | Apple Inc. | Applying neural network language models to weighted finite state transducers for automatic speech recognition |
US10223066B2 (en) | 2015-12-23 | 2019-03-05 | Apple Inc. | Proactive assistance based on dialog communication between devices |
US10446143B2 (en) | 2016-03-14 | 2019-10-15 | Apple Inc. | Identification of voice inputs providing credentials |
US9934775B2 (en) | 2016-05-26 | 2018-04-03 | Apple Inc. | Unit-selection text-to-speech synthesis based on predicted concatenation parameters |
US9972304B2 (en) | 2016-06-03 | 2018-05-15 | Apple Inc. | Privacy preserving distributed evaluation framework for embedded personalized systems |
US11227589B2 (en) | 2016-06-06 | 2022-01-18 | Apple Inc. | Intelligent list reading |
US10249300B2 (en) | 2016-06-06 | 2019-04-02 | Apple Inc. | Intelligent list reading |
US10049663B2 (en) | 2016-06-08 | 2018-08-14 | Apple, Inc. | Intelligent automated assistant for media exploration |
DK179588B1 (en) | 2016-06-09 | 2019-02-22 | Apple Inc. | INTELLIGENT AUTOMATED ASSISTANT IN A HOME ENVIRONMENT |
US10490187B2 (en) | 2016-06-10 | 2019-11-26 | Apple Inc. | Digital assistant providing automated status report |
US10586535B2 (en) | 2016-06-10 | 2020-03-10 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
US10509862B2 (en) | 2016-06-10 | 2019-12-17 | Apple Inc. | Dynamic phrase expansion of language input |
US10192552B2 (en) | 2016-06-10 | 2019-01-29 | Apple Inc. | Digital assistant providing whispered speech |
US10067938B2 (en) | 2016-06-10 | 2018-09-04 | Apple Inc. | Multilingual word prediction |
DK179343B1 (en) | 2016-06-11 | 2018-05-14 | Apple Inc | Intelligent task discovery |
DK179415B1 (en) | 2016-06-11 | 2018-06-14 | Apple Inc | Intelligent device arbitration and control |
DK201670540A1 (en) | 2016-06-11 | 2018-01-08 | Apple Inc | Application integration with a digital assistant |
DK179049B1 (en) | 2016-06-11 | 2017-09-18 | Apple Inc | Data driven natural language event detection and classification |
US10474753B2 (en) | 2016-09-07 | 2019-11-12 | Apple Inc. | Language identification using recurrent neural networks |
US10043516B2 (en) | 2016-09-23 | 2018-08-07 | Apple Inc. | Intelligent automated assistant |
US11281993B2 (en) | 2016-12-05 | 2022-03-22 | Apple Inc. | Model and ensemble compression for metric learning |
US10593346B2 (en) | 2016-12-22 | 2020-03-17 | Apple Inc. | Rank-reduced token representation for automatic speech recognition |
US11204787B2 (en) | 2017-01-09 | 2021-12-21 | Apple Inc. | Application integration with a digital assistant |
DK201770383A1 (en) | 2017-05-09 | 2018-12-14 | Apple Inc. | USER INTERFACE FOR CORRECTING RECOGNITION ERRORS |
US10417266B2 (en) | 2017-05-09 | 2019-09-17 | Apple Inc. | Context-aware ranking of intelligent response suggestions |
DK201770439A1 (en) | 2017-05-11 | 2018-12-13 | Apple Inc. | Offline personal assistant |
US10726832B2 (en) | 2017-05-11 | 2020-07-28 | Apple Inc. | Maintaining privacy of personal information |
US10395654B2 (en) | 2017-05-11 | 2019-08-27 | Apple Inc. | Text normalization based on a data-driven learning network |
DK201770428A1 (en) | 2017-05-12 | 2019-02-18 | Apple Inc. | LOW-LATENCY INTELLIGENT AUTOMATED ASSISTANT |
DK179745B1 (en) | 2017-05-12 | 2019-05-01 | Apple Inc. | SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT |
DK179496B1 (en) | 2017-05-12 | 2019-01-15 | Apple Inc. | USER-SPECIFIC Acoustic Models |
US11301477B2 (en) | 2017-05-12 | 2022-04-12 | Apple Inc. | Feedback analysis of a digital assistant |
DK201770431A1 (en) | 2017-05-15 | 2018-12-20 | Apple Inc. | Optimizing dialogue policy decisions for digital assistants using implicit feedback |
DK201770432A1 (en) | 2017-05-15 | 2018-12-21 | Apple Inc. | Hierarchical belief states for digital assistants |
US10311144B2 (en) | 2017-05-16 | 2019-06-04 | Apple Inc. | Emoji word sense disambiguation |
US20180336892A1 (en) | 2017-05-16 | 2018-11-22 | Apple Inc. | Detecting a trigger of a digital assistant |
US10403278B2 (en) | 2017-05-16 | 2019-09-03 | Apple Inc. | Methods and systems for phonetic matching in digital assistant services |
US20180336275A1 (en) | 2017-05-16 | 2018-11-22 | Apple Inc. | Intelligent automated assistant for media exploration |
DK179549B1 (en) | 2017-05-16 | 2019-02-12 | Apple Inc. | FAR-FIELD EXTENSION FOR DIGITAL ASSISTANT SERVICES |
US10489087B2 (en) | 2017-05-24 | 2019-11-26 | International Business Machines Corporation | Using a space release data structure to indicate tracks to release for a space release command to release space of tracks in a consistency group being formed |
US10528256B2 (en) | 2017-05-24 | 2020-01-07 | International Business Machines Corporation | Processing a space release command to free release space in a consistency group |
US10896097B1 (en) * | 2017-05-25 | 2021-01-19 | Palantir Technologies Inc. | Approaches for backup and restoration of integrated databases |
US10657328B2 (en) | 2017-06-02 | 2020-05-19 | Apple Inc. | Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling |
US20190034475A1 (en) * | 2017-07-28 | 2019-01-31 | Enigma Technologies, Inc. | System and method for detecting duplicate data records |
US10445429B2 (en) | 2017-09-21 | 2019-10-15 | Apple Inc. | Natural language understanding using vocabularies with compressed serialized tries |
US10755051B2 (en) | 2017-09-29 | 2020-08-25 | Apple Inc. | Rule-based natural language processing |
US10789002B1 (en) * | 2017-10-23 | 2020-09-29 | EMC IP Holding Company LLC | Hybrid data deduplication for elastic cloud storage devices |
US10636424B2 (en) | 2017-11-30 | 2020-04-28 | Apple Inc. | Multi-turn canned dialog |
US11086823B2 (en) | 2017-12-28 | 2021-08-10 | Dropbox, Inc. | File deduplication using signature information |
US10733982B2 (en) | 2018-01-08 | 2020-08-04 | Apple Inc. | Multi-directional dialog |
US10733375B2 (en) | 2018-01-31 | 2020-08-04 | Apple Inc. | Knowledge-based framework for improving natural language understanding |
US10789959B2 (en) | 2018-03-02 | 2020-09-29 | Apple Inc. | Training speaker recognition models for digital assistants |
US10592604B2 (en) | 2018-03-12 | 2020-03-17 | Apple Inc. | Inverse text normalization for automatic speech recognition |
US10818288B2 (en) | 2018-03-26 | 2020-10-27 | Apple Inc. | Natural assistant interaction |
US10909331B2 (en) | 2018-03-30 | 2021-02-02 | Apple Inc. | Implicit identification of translation payload with neural machine translation |
US11145294B2 (en) | 2018-05-07 | 2021-10-12 | Apple Inc. | Intelligent automated assistant for delivering content from user experiences |
US10928918B2 (en) | 2018-05-07 | 2021-02-23 | Apple Inc. | Raise to speak |
US10984780B2 (en) | 2018-05-21 | 2021-04-20 | Apple Inc. | Global semantic word embeddings using bi-directional recurrent neural networks |
US10892996B2 (en) | 2018-06-01 | 2021-01-12 | Apple Inc. | Variable latency device coordination |
US11386266B2 (en) | 2018-06-01 | 2022-07-12 | Apple Inc. | Text correction |
DK180639B1 (en) | 2018-06-01 | 2021-11-04 | Apple Inc | DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT |
DK179822B1 (da) | 2018-06-01 | 2019-07-12 | Apple Inc. | Voice interaction at a primary device to access call functionality of a companion device |
DK201870355A1 (en) | 2018-06-01 | 2019-12-16 | Apple Inc. | VIRTUAL ASSISTANT OPERATION IN MULTI-DEVICE ENVIRONMENTS |
US11076039B2 (en) | 2018-06-03 | 2021-07-27 | Apple Inc. | Accelerated task performance |
US11010561B2 (en) | 2018-09-27 | 2021-05-18 | Apple Inc. | Sentiment prediction from textual data |
US11462215B2 (en) | 2018-09-28 | 2022-10-04 | Apple Inc. | Multi-modal inputs for voice commands |
US11170166B2 (en) | 2018-09-28 | 2021-11-09 | Apple Inc. | Neural typographical error modeling via generative adversarial networks |
US10839159B2 (en) | 2018-09-28 | 2020-11-17 | Apple Inc. | Named entity normalization in a spoken dialog system |
US11475898B2 (en) | 2018-10-26 | 2022-10-18 | Apple Inc. | Low-latency multi-speaker speech recognition |
US10649676B1 (en) | 2018-12-28 | 2020-05-12 | Microsoft Technology Licensing, Llc | Memory compression for immutable data structures |
US11638059B2 (en) | 2019-01-04 | 2023-04-25 | Apple Inc. | Content playback on multiple devices |
US11348573B2 (en) | 2019-03-18 | 2022-05-31 | Apple Inc. | Multimodality in digital assistant systems |
CN111814003B (zh) * | 2019-04-12 | 2024-04-23 | 伊姆西Ip控股有限责任公司 | 用于建立元数据索引的方法、电子设备和计算机程序产品 |
DK201970509A1 (en) | 2019-05-06 | 2021-01-15 | Apple Inc | Spoken notifications |
US11423908B2 (en) | 2019-05-06 | 2022-08-23 | Apple Inc. | Interpreting spoken requests |
US11475884B2 (en) | 2019-05-06 | 2022-10-18 | Apple Inc. | Reducing digital assistant latency when a language is incorrectly determined |
US11307752B2 (en) | 2019-05-06 | 2022-04-19 | Apple Inc. | User configurable task triggers |
US11140099B2 (en) | 2019-05-21 | 2021-10-05 | Apple Inc. | Providing message response suggestions |
US11289073B2 (en) | 2019-05-31 | 2022-03-29 | Apple Inc. | Device text to speech |
DK201970511A1 (en) | 2019-05-31 | 2021-02-15 | Apple Inc | Voice identification in digital assistant systems |
US11496600B2 (en) | 2019-05-31 | 2022-11-08 | Apple Inc. | Remote execution of machine-learned models |
DK180129B1 (en) | 2019-05-31 | 2020-06-02 | Apple Inc. | USER ACTIVITY SHORTCUT SUGGESTIONS |
US11360641B2 (en) | 2019-06-01 | 2022-06-14 | Apple Inc. | Increasing the relevance of new available information |
WO2021056255A1 (en) | 2019-09-25 | 2021-04-01 | Apple Inc. | Text detection using global geometry estimators |
US11429573B2 (en) | 2019-10-16 | 2022-08-30 | Dell Products L.P. | Data deduplication system |
CN114610227B (zh) * | 2020-12-09 | 2024-03-19 | 戴尔产品有限公司 | 将存储系统集群化以共享用于去重的数据模式 |
US11360867B1 (en) | 2021-03-31 | 2022-06-14 | Netapp, Inc. | Re-aligning data replication configuration of primary and secondary data serving entities of a cross-site storage solution after a failover event |
US11709743B2 (en) | 2021-03-31 | 2023-07-25 | Netapp, Inc. | Methods and systems for a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system |
US11550679B2 (en) | 2021-03-31 | 2023-01-10 | Netapp, Inc. | Methods and systems for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system |
US11934670B2 (en) | 2021-03-31 | 2024-03-19 | Netapp, Inc. | Performing various operations at the granularity of a consistency group within a cross-site storage solution |
US11481139B1 (en) | 2021-03-31 | 2022-10-25 | Netapp, Inc. | Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity |
US11740811B2 (en) | 2021-03-31 | 2023-08-29 | Netapp, Inc. | Reseeding a mediator of a cross-site storage solution |
US11409622B1 (en) | 2021-04-23 | 2022-08-09 | Netapp, Inc. | Methods and systems for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system without using an external mediator |
US11893261B2 (en) | 2021-05-05 | 2024-02-06 | Netapp, Inc. | Usage of OP logs to synchronize across primary and secondary storage clusters of a cross-site distributed storage system and lightweight OP logging |
US11537314B1 (en) | 2021-10-07 | 2022-12-27 | Netapp, Inc. | Resynchronization of individual volumes of a consistency group (CG) within a cross-site storage solution while maintaining synchronization of other volumes of the CG |
US11892982B2 (en) | 2021-10-20 | 2024-02-06 | Netapp, Inc. | Facilitating immediate performance of volume resynchronization with the use of passive cache entries |
US11907562B2 (en) | 2022-07-11 | 2024-02-20 | Netapp, Inc. | Methods and storage nodes to decrease delay in resuming input output (I/O) operations after a non-disruptive event for a storage object of a distributed storage system by utilizing asynchronous inflight replay of the I/O operations |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6438562B1 (en) * | 1999-08-24 | 2002-08-20 | Oracle Corporation | Parallel index maintenance |
US20050193024A1 (en) * | 2004-02-27 | 2005-09-01 | Beyer Kevin S. | Asynchronous peer-to-peer data replication |
Family Cites Families (67)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2583010B2 (ja) * | 1993-01-07 | 1997-02-19 | インターナショナル・ビジネス・マシーンズ・コーポレイション | 多層インデックス構造におけるローカルインデックステーブル及び大域インデックステーブルの間の一貫性を維持する方法 |
US6192365B1 (en) * | 1995-07-20 | 2001-02-20 | Novell, Inc. | Transaction log management in a disconnectable computer and network |
US5897664A (en) * | 1996-07-01 | 1999-04-27 | Sun Microsystems, Inc. | Multiprocessor system having mapping table in each node to map global physical addresses to local physical addresses of page copies |
US6412017B1 (en) * | 1996-07-01 | 2002-06-25 | Microsoft Corporation | Urgent replication facility |
US5812773A (en) * | 1996-07-12 | 1998-09-22 | Microsoft Corporation | System and method for the distribution of hierarchically structured data |
CN1232914C (zh) * | 2000-04-26 | 2005-12-21 | 计算机合作者思维公司 | 在分布计算机系统中保持数据完整性的方法与设备 |
JP4096147B2 (ja) * | 2000-10-25 | 2008-06-04 | 株式会社日立製作所 | 分散型計算機システムにおける重複配置複製データの複製方式 |
US7231391B2 (en) * | 2001-02-06 | 2007-06-12 | Quest Software, Inc. | Loosely coupled database clusters with client connection fail-over |
US20020161907A1 (en) * | 2001-04-25 | 2002-10-31 | Avery Moon | Adaptive multi-protocol communications system |
US6988124B2 (en) * | 2001-06-06 | 2006-01-17 | Microsoft Corporation | Locating potentially identical objects across multiple computers based on stochastic partitioning of workload |
US7451217B2 (en) | 2002-12-19 | 2008-11-11 | International Business Machines Corporation | Method and system for peer-to-peer authorization |
US7573500B2 (en) * | 2003-03-24 | 2009-08-11 | Sensormatic Electronics Corporation | System and method for communicating data in a video system |
US7296008B2 (en) * | 2004-08-24 | 2007-11-13 | Symantec Operating Corporation | Generation and use of a time map for accessing a prior image of a storage device |
US7685384B2 (en) * | 2004-02-06 | 2010-03-23 | Globalscape, Inc. | System and method for replicating files in a computer network |
US7200604B2 (en) * | 2004-02-17 | 2007-04-03 | Hewlett-Packard Development Company, L.P. | Data de-duplication |
US7206790B2 (en) * | 2004-07-13 | 2007-04-17 | Hitachi, Ltd. | Data management system |
US7778984B2 (en) * | 2004-11-19 | 2010-08-17 | Microsoft Corporation | System and method for a distributed object store |
JP2006185041A (ja) * | 2004-12-27 | 2006-07-13 | Matsushita Electric Ind Co Ltd | コンテンツ分散配置システム、端末及びコンテンツ分散配置システムの動作方法 |
US7571168B2 (en) * | 2005-07-25 | 2009-08-04 | Parascale, Inc. | Asynchronous file replication and migration in a storage network |
US7584338B1 (en) * | 2005-09-27 | 2009-09-01 | Data Domain, Inc. | Replication of deduplicated storage system |
US7788223B2 (en) * | 2005-12-05 | 2010-08-31 | Microsoft Corporation | Resource freshness and replication |
JP4920979B2 (ja) | 2006-01-25 | 2012-04-18 | 株式会社日立製作所 | ストレージ装置及びその制御方法 |
US7921077B2 (en) * | 2006-06-29 | 2011-04-05 | Netapp, Inc. | System and method for managing data deduplication of storage systems utilizing persistent consistency point images |
US7747584B1 (en) * | 2006-08-22 | 2010-06-29 | Netapp, Inc. | System and method for enabling de-duplication in a storage system architecture |
US8635194B2 (en) * | 2006-10-19 | 2014-01-21 | Oracle International Corporation | System and method for data compression |
JP4945232B2 (ja) * | 2006-12-21 | 2012-06-06 | 株式会社日立製作所 | アクセス制御方法、計算機システム、及びオブジェクト複製プログラム |
US7840537B2 (en) * | 2006-12-22 | 2010-11-23 | Commvault Systems, Inc. | System and method for storing redundant information |
JP4951331B2 (ja) * | 2006-12-26 | 2012-06-13 | 株式会社日立製作所 | ストレージシステム |
US8719375B2 (en) * | 2007-03-22 | 2014-05-06 | Microsoft Corporation | Remote data access techniques for portable devices |
US7873809B2 (en) * | 2007-03-29 | 2011-01-18 | Hitachi, Ltd. | Method and apparatus for de-duplication after mirror operation |
US8768895B2 (en) * | 2007-04-11 | 2014-07-01 | Emc Corporation | Subsegmenting for efficient storage, resemblance determination, and transmission |
US8204866B2 (en) * | 2007-05-18 | 2012-06-19 | Microsoft Corporation | Leveraging constraints for deduplication |
US8315984B2 (en) * | 2007-05-22 | 2012-11-20 | Netapp, Inc. | System and method for on-the-fly elimination of redundant data |
US7827168B2 (en) * | 2007-05-30 | 2010-11-02 | Red Hat, Inc. | Index clustering for full text search engines |
US7669023B2 (en) * | 2007-07-10 | 2010-02-23 | Hitachi, Ltd. | Power efficient storage with data de-duplication |
US8793226B1 (en) * | 2007-08-28 | 2014-07-29 | Netapp, Inc. | System and method for estimating duplicate data |
WO2009032710A2 (en) * | 2007-08-29 | 2009-03-12 | Nirvanix, Inc. | Filing system and method for data files stored in a distributed communications network |
US7822939B1 (en) * | 2007-09-25 | 2010-10-26 | Emc Corporation | Data de-duplication using thin provisioning |
US7870409B2 (en) * | 2007-09-26 | 2011-01-11 | Hitachi, Ltd. | Power efficient data storage with data de-duplication |
US8548953B2 (en) * | 2007-11-12 | 2013-10-01 | F5 Networks, Inc. | File deduplication using storage tiers |
US7870105B2 (en) * | 2007-11-20 | 2011-01-11 | Hitachi, Ltd. | Methods and apparatus for deduplication in storage system |
US7949622B2 (en) * | 2007-12-13 | 2011-05-24 | Yahoo! Inc. | System and method for generating a classifier model for classifying web content |
US8135918B1 (en) * | 2007-12-31 | 2012-03-13 | Emc Corporation | Data de-duplication for iSCSI |
US8190835B1 (en) * | 2007-12-31 | 2012-05-29 | Emc Corporation | Global de-duplication in shared architectures |
US7783604B1 (en) * | 2007-12-31 | 2010-08-24 | Emc Corporation | Data de-duplication and offsite SaaS backup and archiving |
US20090204636A1 (en) * | 2008-02-11 | 2009-08-13 | Microsoft Corporation | Multimodal object de-duplication |
US7962706B2 (en) * | 2008-02-14 | 2011-06-14 | Quantum Corporation | Methods and systems for improving read performance in data de-duplication storage |
US7814074B2 (en) * | 2008-03-14 | 2010-10-12 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US7984022B2 (en) * | 2008-04-18 | 2011-07-19 | International Business Machines Corporation | Space recovery with storage management coupled with a deduplicating storage system |
US7908436B1 (en) * | 2008-04-25 | 2011-03-15 | Netapp, Inc. | Deduplication of data on disk devices using low-latency random read memory |
US8346730B2 (en) * | 2008-04-25 | 2013-01-01 | Netapp. Inc. | Deduplication of data on disk devices based on a threshold number of sequential blocks |
US7930306B2 (en) * | 2008-04-30 | 2011-04-19 | Msc Intellectual Properties B.V. | System and method for near and exact de-duplication of documents |
US8645333B2 (en) * | 2008-05-29 | 2014-02-04 | International Business Machines Corporation | Method and apparatus to minimize metadata in de-duplication |
US7996371B1 (en) * | 2008-06-10 | 2011-08-09 | Netapp, Inc. | Combining context-aware and context-independent data deduplication for optimal space savings |
US8108353B2 (en) * | 2008-06-11 | 2012-01-31 | International Business Machines Corporation | Method and apparatus for block size optimization in de-duplication |
US8682850B2 (en) * | 2008-06-11 | 2014-03-25 | International Business Machines Corporation | Method of enhancing de-duplication impact by preferential selection of master copy to be retained |
US8484162B2 (en) * | 2008-06-24 | 2013-07-09 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US8335776B2 (en) * | 2008-07-02 | 2012-12-18 | Commvault Systems, Inc. | Distributed indexing system for data storage |
US8788466B2 (en) * | 2008-08-05 | 2014-07-22 | International Business Machines Corporation | Efficient transfer of deduplicated data |
US7814149B1 (en) * | 2008-09-29 | 2010-10-12 | Symantec Operating Corporation | Client side data deduplication |
US8495032B2 (en) * | 2008-10-01 | 2013-07-23 | International Business Machines Corporation | Policy based sharing of redundant data across storage pools in a deduplicating system |
JP5579195B2 (ja) * | 2008-12-22 | 2014-08-27 | グーグル インコーポレイテッド | 複製されたコンテンツアドレス可能ストレージクラスタのための非同期分散型重複排除 |
US8356017B2 (en) * | 2009-08-11 | 2013-01-15 | International Business Machines Corporation | Replication of deduplicated data |
US8868508B2 (en) * | 2010-02-09 | 2014-10-21 | Google Inc. | Storage of data in a distributed storage system |
CA2912394A1 (en) * | 2013-05-14 | 2014-11-20 | Actifio, Inc. | Efficient data replication and garbage collection predictions |
US9928248B2 (en) * | 2013-07-08 | 2018-03-27 | International Business Machines Corporation | Self-healing by hash-based deduplication |
US9817865B2 (en) * | 2015-12-07 | 2017-11-14 | International Business Machines Corporation | Direct lookup for identifying duplicate data in a data deduplication system |
-
2009
- 2009-12-22 JP JP2011542576A patent/JP5579195B2/ja active Active
- 2009-12-22 BR BRPI0922990-6A patent/BRPI0922990B1/pt active IP Right Grant
- 2009-12-22 AU AU2009330073A patent/AU2009330073B2/en active Active
- 2009-12-22 EP EP09799831.4A patent/EP2368199B1/en active Active
- 2009-12-22 CN CN201410306908.5A patent/CN104166673B/zh active Active
- 2009-12-22 US US12/644,693 patent/US8712974B2/en active Active
- 2009-12-22 WO PCT/US2009/069234 patent/WO2010075407A1/en active Application Filing
- 2009-12-22 DE DE202009019139.7U patent/DE202009019139U1/de not_active Expired - Lifetime
- 2009-12-22 CN CN200980156970.8A patent/CN102317938B/zh active Active
- 2009-12-22 CA CA2747746A patent/CA2747746C/en active Active
-
2014
- 2014-03-25 JP JP2014061617A patent/JP5902222B2/ja active Active
- 2014-04-29 US US14/265,298 patent/US20140236888A1/en not_active Abandoned
-
2016
- 2016-01-13 US US14/995,171 patent/US10291699B2/en active Active
-
2019
- 2019-04-22 US US16/390,613 patent/US11943290B2/en active Active
-
2024
- 2024-03-07 US US18/598,041 patent/US20240251012A1/en active Pending
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6438562B1 (en) * | 1999-08-24 | 2002-08-20 | Oracle Corporation | Parallel index maintenance |
US20050193024A1 (en) * | 2004-02-27 | 2005-09-01 | Beyer Kevin S. | Asynchronous peer-to-peer data replication |
Non-Patent Citations (5)
Title |
---|
CHANG ET AL.: "Bigtable: A Distributed Storage System for Structured Data", PROC. OF THE 7TH OSDI, November 2006 (2006-11-01), pages 205 - 218, XP001544719 |
GEER D: "Reducing the Storage Burden via Data Deduplication", COMPUTER, IEEE SERVICE CENTER, LOS ALAMITOS, CA, US, vol. 41, no. 12, 1 December 2008 (2008-12-01), pages 15 - 17, XP011249422, ISSN: 0018-9162 * |
MIN-YAN WANG ET AL: "The Research of Web Page De-duplication Based on Web Pages Reshipment Statement", DATABASE TECHNOLOGY AND APPLICATIONS, 2009 FIRST INTERNATIONAL WORKSHOP ON, IEEE, PISCATAWAY, NJ, USA, 25 April 2009 (2009-04-25), pages 271 - 274, XP031515216, ISBN: 978-0-7695-3604-0 * |
RICHARD G GUY ET AL: "Implementation of the Ficus Replicated File System", PROCEEDINGS OF THE SUMMER USENIX CONFERENCE, XX, XX, 30 June 1990 (1990-06-30), pages 63 - 71, XP002234187 * |
WIESMANN M ET AL: "Database replication techniques: a three parameter classification", RELIABLE DISTRIBUTED SYSTEMS, 2000. SRDS-2000. PROCEEDINGS THE 19TH IE EE SYMPOSIUM ON NURNBERG, GERMANY 16-18 OCT. 2000, LOS ALAMITOS, CA, USA,IEEE COMPUT. SOC, US, 16 October 2000 (2000-10-16), pages 206 - 215, XP010523961, ISBN: 978-0-7695-0543-5 * |
Also Published As
Publication number | Publication date |
---|---|
EP2368199B1 (en) | 2018-10-31 |
CA2747746C (en) | 2015-08-11 |
US8712974B2 (en) | 2014-04-29 |
DE202009019139U1 (de) | 2017-01-20 |
US11943290B2 (en) | 2024-03-26 |
US20140236888A1 (en) | 2014-08-21 |
BRPI0922990B1 (pt) | 2020-11-10 |
AU2009330073B2 (en) | 2012-11-15 |
CN102317938B (zh) | 2014-07-30 |
CN104166673B (zh) | 2017-09-19 |
JP2012513640A (ja) | 2012-06-14 |
US10291699B2 (en) | 2019-05-14 |
EP2368199A1 (en) | 2011-09-28 |
CN104166673A (zh) | 2014-11-26 |
US20100161554A1 (en) | 2010-06-24 |
US20240251012A1 (en) | 2024-07-25 |
JP5902222B2 (ja) | 2016-04-13 |
CN102317938A (zh) | 2012-01-11 |
JP2014139824A (ja) | 2014-07-31 |
AU2009330073A1 (en) | 2011-07-07 |
CA2747746A1 (en) | 2010-07-01 |
JP5579195B2 (ja) | 2014-08-27 |
US20160134696A1 (en) | 2016-05-12 |
BRPI0922990A2 (pt) | 2016-01-26 |
US20190268411A1 (en) | 2019-08-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11943290B2 (en) | Asynchronous distributed de-duplication for replicated content addressable storage clusters | |
CA2758518C (en) | Asynchronous distributed object uploading for replicated content addressable storage clusters | |
US9081841B2 (en) | Asynchronous distributed garbage collection for replicated storage clusters | |
US11663166B2 (en) | Post-processing global deduplication algorithm for scaled-out deduplication file system | |
Xu et al. | YuruBackup: a space-efficient and highly scalable incremental backup system in the cloud | |
AU2013209349B2 (en) | Asynchronous distributed object uploading for replicated content addressable storage clusters | |
Pandey et al. | A review on google file system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200980156970.8 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09799831 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2747746 Country of ref document: CA Ref document number: 2009330073 Country of ref document: AU |
|
ENP | Entry into the national phase |
Ref document number: 2011542576 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2009799831 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2009330073 Country of ref document: AU Date of ref document: 20091222 Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: PI0922990 Country of ref document: BR Kind code of ref document: A2 Effective date: 20110617 |