TW202137027A - Data storage system and global deduplication method thereof - Google Patents

Abstract

A data storage system and a global deduplication method thereof are provided. The data storage system includes multiple storage devices and a dispatch device. The dispatch device divides the original data corresponding to a data writing request into at least one data chunk. The dispatch device performs a summary calculation on one data chunk to generate a representative value. The dispatch device performs a distribution calculation on the representative value to determine the destination location corresponding to the representative value. The dispatch device transmits the data chunk and the representative value to at least one destination storage device among the storage devices through a communication network according to the destination location. The at least one destination storage device checks the representative value to determine whether to store the data chunk in the storage space of the at least one destination storage device.

Description

Data storage system and its global deduplication method

The present invention relates to a data storage technology, and particularly relates to a data storage system and its global deduplication method.

The distributed storage system (data storage system) includes multiple object storage devices. These object storage devices can be connected to each other via a communication network, where the communication network includes a local area network and/or the Internet. Multiple user devices can store data in a distributed storage system via the network. In any case, the data stored in the distributed storage system may be duplicated. In order to store a large amount of data more effectively in a distributed storage system, a data reduction technology, such as deduplication, is needed. Deduplication technology can eliminate duplicates of duplicate data to reduce the amount of stored data.

An example of a distributed storage system is Ceph. The well-known Ceph is an open-source software storage platform that implements storage of objects in a single distributed computer cluster. Due to the data processing process and additional metadata management of the distributed storage system, it is difficult for the distributed storage system to perform global deduplication. For example, Ceph does not support global deduplication.

It should be noted that the content of the "prior art" paragraph is used to help understand the present invention. Part of the content (or all of the content) disclosed in the "Prior Art" paragraph may not be the conventional technology known to those with ordinary knowledge in the technical field. The content disclosed in the "prior art" paragraph does not mean that the content has been known to those with ordinary knowledge in the technical field before the application of the present invention.

The present invention provides a data storage system and a global deduplication method thereof, which can effectively deduplication in a distributed storage environment.

In an embodiment of the present invention, the aforementioned data storage system includes a plurality of storage devices and a dispatch device. These storage devices are suitable for coupling to a communication network. The dispatching device is adapted to receive data write requests. The dispatching device is configured to cut the original data corresponding to the data write request into at least one data block. The allocating device performs summary calculation on a current data block in the at least one data block to generate a representative value corresponding to the current data block. The allocating device performs a first distribution calculation on the representative value to determine the destination location corresponding to the representative value. The dispatching device transmits the current data block and the representative value to at least one of the storage devices through the communication network according to the destination location. The at least one destination storage device checks the representative value to determine whether to store the current data block in the storage space of the at least one destination storage device.

In an embodiment of the present invention, the aforementioned global deduplication method includes: receiving a data write request by a dispatching device; cutting the original data corresponding to the data writing request into at least one data block by the dispatching device; A current data block in the at least one data block performs summary calculation to generate a representative value corresponding to the current data block; the dispatching device performs a first distribution calculation on the representative value to determine the destination location corresponding to the representative value; The dispatching device transmits the current data block and the representative value to at least one destination storage device among the plurality of storage devices through the communication network according to the destination location; and the at least one destination storage device checks the representative value to determine whether to store the current data block Into the storage space of the at least one destination storage device.

Based on the above, the data storage system and its global deduplication method described in the embodiments of the present invention can be applied to a distributed storage system. The dispatch device can generate the representative value corresponding to the current data block, and determine the destination location based on the representative value. According to the destination location, the allocating device can transmit the current data block and representative value to at least one destination storage device among the multiple storage devices through a communication network (such as a local area network or other network). In other words, the same data block will be sent to the same storage device. The destination storage device can check the representative value to determine whether to store the current data block in its local storage space. Therefore, the data storage system can effectively de-duplicate in a distributed storage environment.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

The term "coupling (or connection)" used in the full text of the description of this case (including the scope of the patent application) can refer to any direct or indirect connection means. For example, if it is described in the text that the first device is coupled (or connected) to the second device, it should be interpreted as that the first device can be directly connected to the second device, or the first device can be connected through other devices or some This kind of connection means is indirectly connected to the second device. The terms "first" and "second" mentioned in the full text of the description of this case (including the scope of the patent application) are used to name the element (element), or to distinguish different embodiments or ranges, and are not used to limit the number of elements The upper or lower limit of is not used to limit the order of components. In addition, wherever possible, elements/components/steps with the same reference numbers in the drawings and embodiments represent the same or similar parts. Elements/components/steps that use the same reference numerals or use the same terms in different embodiments may refer to related descriptions.

FIG. 1 is a schematic diagram of a circuit block of a data storage system 100 according to an embodiment of the present invention. The user device 10 can send data write requests, data read requests, and/or other requests to the data storage system 100 via the communication network 20. The data storage system 100 can provide cloud storage services to one or more user devices 10 via the communication network 20. The user device 10 may be a mobile phone, a tablet computer, a notebook computer, a personal computer, and/or other electronic devices. The communication network 20 may include a local area network, the Internet, and/or other communication networks.

In the embodiment shown in FIG. 1, the data storage system 100 includes a dispatch device 110, a storage device 120_1, a storage device 120_2, a storage device 120_3, a storage device 120_4, a storage device 120_5, ..., a storage device 120_n. The dispatching device 110 and the storage devices 120_1 to 120_n are suitable for coupling to the communication network 20. The number n of storage devices 120_1 to 120_n can be determined according to design requirements. For example, in an embodiment, the number n may be 2.

The storage devices 120_1 to 120_n may be multiple electronic devices independent of each other. According to design requirements, any of these storage devices 120_1 to 120_n can be a personal computer, a workstation, a server, a network attached storage (NAS) device, and/or other electronic devices. According to design requirements, in some embodiments, the dispatching device 110 may be another electronic device independent of the storage devices 120_1 to 120_n, and the dispatching device 110 may be a personal computer, a workstation, a server, a NAS device, and/or other Electronic equipment. In other embodiments, at least one of the storage devices 120_1 to 120_n may be selected as the dispatching device 110.

According to different design requirements, the implementation of the blocks of the dispatch device 110 and/or the storage devices 120_1 to 120_n can be hardware, firmware, software, or the aforementioned three A combination of more than one of them. In terms of hardware, the related functions of the blocks of the aforementioned dispatching device 110 and/or storage devices 120_1 to 120_n can be implemented in one or more controllers, microcontrollers, microprocessors, and integrated circuits for special applications. (Application-specific integrated circuit, ASIC), digital signal processor (DSP), Field Programmable Gate Array (FPGA) and/or various logic blocks and modules in other processing units Groups and circuits.

In terms of software form and/or firmware form, the related functions of the aforementioned dispatching device 110 and/or the storage devices 120_1 to 120_n can be implemented as programming codes. For example, general programming languages (such as C, C++ or assembly languages) or other suitable programming languages are used to implement the aforementioned dispatching device 110 and/or the storage devices 120_1 to 120_n. The programming code can be recorded/stored in a recording medium. In some embodiments, the recording medium includes, for example, a read only memory (Read Only Memory, ROM), a storage device, and/or a random access memory (Random Access Memory, RAM). In other embodiments, the recording medium may include "non-transitory computer readable medium". For example, tape, disk, card, semiconductor memory, programmable logic circuit, etc. can be used to implement the non-temporary computer readable medium. A computer, a central processing unit (CPU), a controller, a microcontroller, or a microprocessor can read and execute the programming code from the recording medium, thereby realizing the aforementioned dispatching device 110 and/or Related functions of the storage devices 120_1 to 120_n. Moreover, the programming code can also be provided to the computer (or CPU) via any transmission medium (communication network or broadcast wave, etc.). The communication network is, for example, the Internet, wired communication, wireless communication, or other communication media.

FIG. 2 is a schematic block diagram of a circuit of a data storage system 200 according to another embodiment of the present invention. The user device 10 can send data write requests, data read requests, and/or other requests to the data storage system 200 via the communication network 30. The data storage system 200 can provide cloud storage services to one or more user devices 10 via the communication network 30. The communication network 30 may include a local area network, the Internet, and/or other communication networks. The user device 10 shown in FIG. 2 can refer to the related description of the user device 10 shown in FIG.

In the embodiment shown in FIG. 2, the data storage system 200 includes a dispatching device 110 and storage devices 120_1 to 120_n. The dispatching device 110 and the storage devices 120_1 to 120_n are suitable for coupling to the communication network 20. The communication network 20, the dispatching device 110, and the storage devices 120_1 to 120_n shown in FIG. 2 can refer to the related descriptions of the communication network 20, the dispatching device 110 and the storage devices 120_1 to 120_n shown in FIG.

Fig. 3 is a schematic flowchart of a global de-duplication method according to an embodiment of the present invention. The global deduplication method shown in FIG. 3 can be applied to the data storage system 100 shown in FIG. 1 and/or the data storage system 200 shown in FIG. 2. According to design requirements, readers can refer to FIG. 1 and FIG. 3 to understand the following description content, or refer to FIG. 2 and FIG. 3 to understand the following description content.

In step S310, the dispatching device 110 may receive a data write request from the user device 10. The data write request may include the original key value OID and the original data D. That is, the user device 10 requests to write a piece of original data D into the data storage system.

In step S320, the dispatching device 110 may cut the original data D corresponding to the data write request into one (or more) data chunks. In step S330, the allocating device 110 may perform a summary calculation on a current data block in this (or these) data blocks to generate a representative value corresponding to the current data block. According to design requirements, in some embodiments, the digest calculation may include a Hash algorithm and/or other algorithms, and the representative value may include the hash value of the current data block. For example, the dispatching device 110 may cut the original data D into a data block ChunkA and a data block ChunkB, the dispatching device 110 may perform a hash calculation on the data block ChunkA to generate the representative value HIDa, and the dispatching device 110 may perform a hash calculation on the data block ChunkB performs a hash calculation to generate a representative value HIDb. The dispatching device 110 may record the mapping relationship between the original key value OID, the representative value HIDa, and the representative value HIDb in the lookup table of the dispatching device 110.

In step S340, the allocating device 110 may perform a distribution calculation on the representative value to determine the destination location corresponding to the representative value. In step S350, the allocating device 110 can transmit the current data block and the representative value to at least one of the storage devices 120_1 to 120_n through the communication network 20 according to the destination location. According to design requirements, in some embodiments, the distributed calculation may include the Ceph CRUSH algorithm and/or other algorithms. The Ceph CRUSH algorithm is a conventional technology, so it will not be repeated here. For example, the allocating device 110 may perform CRUSH calculation on the representative value HIDa of the data block ChunkA, and transmit the data block ChunkA to the storage device 120_1 (destination storage device) according to the calculation result (hypothesis). Similarly, the dispatching device 110 can perform CRUSH calculation on the representative value HIDb of the data block ChunkB, and transmit the data block ChunkB to the storage device 120_4 (destination storage device) according to the calculation result (hypothesis).

In step S360, the destination storage device may check the representative value to determine whether to store the current data block in the storage space of the destination storage device. According to design requirements, the storage space may be a hard disk drive (HDD), a solid state drive (SSD), and/or other storage media of the destination storage device. For example, after the dispatch device 110 transmits the data block ChunkA and the representative value HIDa to the storage device 120_1 (destination storage device), the storage device 120_1 can check the representative value HIDa to determine whether to store the data block ChunkA in the storage device 120_1. In storage space. Similarly, after the dispatching device 110 transmits the data block ChunkB and the representative value HIDb to the storage device 120_4 (destination storage device), the storage device 120_4 can check the representative value HIDb to determine whether to store the data block ChunkB in the storage device 120_1. In space.

FIG. 4 is a schematic diagram of the operation flow of a storage device according to an embodiment of the present invention. For step S360 shown in FIG. 3, reference may be made to the related description of FIG. 4. In the embodiment shown in FIG. 4, each of the storage devices 120_1 to 120_n has a look-up table (or a link table). In step S410, the destination storage device may receive the current data block and representative value provided by the dispatching device 110. In step S420, the destination storage device may check whether the lookup table has the representative value. For example, after the dispatch device 110 transmits the data block ChunkA and the representative value HIDa to the storage device 120_1 (destination storage device), the storage device 120_1 can check the lookup table of the storage device 120_1 for the representative value HIDa.

When the look-up table has the representative value (that is, the judgment result of step S430 is "Yes"), the destination storage device may perform steps S440, S450, and S460. In step S440, the destination storage device will give up storing the current data block provided by the dispatching device 110 in the storage space, that is, discard the current data block provided by the dispatching device 110. In step S450, the destination storage device may increase the reference times value corresponding to the representative value. In step S460, the destination storage device may update the reference count value to the look-up table. For example, in the case where the look-up table of the storage device 120_1 (destination storage device) has the representative value HIDa, the storage device 120_1 may discard the current data block ChunkA provided by the dispatching device 110, and correspond the representative value HIDa The reference count value RefCnt is incremented by 1, and the reference count value RefCnt is updated to the look-up table of the storage device 120_1.

When the look-up table does not have the representative value (that is, the judgment result of step S430 is “No”), the destination storage device may perform steps S470, S480, and S490. In step S470, the destination storage device may store the current data block provided by the dispatching device 110 in the physical address of the storage space. In step S480, the destination storage device may set the reference count value corresponding to the representative value as an initial value (for example, 0). In step S490, the destination storage device may record the representative value, the physical address, and the reference count value in the lookup table. For example, suppose that the length of the current data block ChunkA is L. In the case that the look-up table of the storage device 120_1 (destination storage device) does not have the representative value HIDa, the storage device 120_1 can store the current data block ChunkA provided by the dispatch device 110 in a physical location of the storage space of the storage device 120_1 Address FADD, reset the reference count value RefCnt corresponding to the representative value HIDa to 0, and record the representative value HIDa, the physical address FADD, the length L, and the reference count value RefCnt in the look-up table of the storage device 120_1.

FIG. 5 is a circuit block diagram illustrating the dispatching device 110 shown in FIG. 1 or FIG. 2 according to an embodiment of the present invention. According to design requirements, the embodiment described in FIG. 5 can refer to the related descriptions of FIG. 3 and (or) FIG. 4. In the embodiment shown in FIG. 5, the dispatching device 110 includes a main dispatching device 111 and a backup dispatching device 112. The main dispatching device 111 and the backup dispatching device 112 are adapted to be coupled to the communication network 20. The main dispatching device 111 and the backup dispatching device 112 are two independent electronic devices. According to design requirements, any one of the main dispatching device 111 and the backup dispatching device 112 can be a personal computer, a workstation, a server, a NAS device, and/or other electronic devices. According to design requirements, in some embodiments, the main dispatching device 111 (or the backup dispatching device 112) may be another electronic device independent of the storage devices 120_1 to 120_n. In other embodiments, at least one of the storage devices 120_1 to 120_n may be selected as the main dispatching device 111 (or the backup dispatching device 112).

According to different design requirements, the implementation of the blocks of the primary dispatching device 111 and/or the redundant dispatching device 112 can be hardware, firmware, software (ie, programs), or a combination of many of the foregoing three. form. In the form of software and/or firmware, the related functions of the above-mentioned primary dispatching device 111 and/or the backup dispatching device 112 can be implemented as programming codes. For example, a general programming language (for example, C, C++, or assembly language) or other suitable programming languages are used to implement the above-mentioned main dispatching device 111 and/or the backup dispatching device 112. The programming code can be recorded/stored in a recording medium. A computer, CPU, controller, microcontroller, or microprocessor can read and execute the programming code from the recording medium, thereby realizing the above-mentioned main dispatching device 111 and (or) the related functions of the backup dispatching device 112 . Moreover, the programming code can also be provided to the computer (or CPU) via any transmission medium (communication network or broadcast wave, etc.).

FIG. 6 is a schematic diagram illustrating a write operation flow of the master dispatching device 111 shown in FIG. 5 according to an embodiment of the present invention. The functions of the backup dispatching device 112 shown in FIG. 5 can be deduced by analogy with reference to the relevant description of the main dispatching device 111. Please refer to Figure 5 and Figure 6. In step S610, the master dispatching device 111 may receive a data write request from the user device 10. The data write request may include the original key value OID and the original data D.

In step S620, the main dispatching device 111 may cut the original data D corresponding to the data write request into one (or more) data chunks. In step S630, the main dispatching device 111 may perform a summary calculation on a current data block in this (or these) data blocks to generate a representative value corresponding to the current data block. In step S640, the main dispatching device 111 may record the mapping relationship between the original key value and the representative value in the lookup table of the main dispatching device 111. For example, the main dispatching device 111 may cut the original data D into a data block ChunkA and a data block ChunkB, the main dispatching device 111 may perform a hash calculation on the data block ChunkA to generate a representative value HIDa, and the main dispatching device 111 may Hash calculation is performed on the data block ChunkB to generate the representative value HIDb. The main dispatching device 111 may record the mapping relationship between the original key value OID, the representative value HIDa and the representative value HIDb in the lookup table of the main dispatching device 111.

In step S650, the main dispatching device 111 may perform a distribution calculation on the representative value to determine the destination location corresponding to the representative value. In step S660, the master dispatching device 111 may transmit the current data block and representative value to multiple destination storage devices among the storage devices 120_1 to 120_n through the communication network 20 according to the destination location. For example, the main dispatching device 111 may perform CRUSH calculation on the representative value HIDa of the data block ChunkA, and transmit the representative value HIDa and the data block ChunkA to the storage devices 120_1 and 120_2 (destination storage devices) according to the calculation result (hypothesis). The same data block ChunkA is transferred to the storage devices 120_1 and 120_2 to realize the replication function. Similarly, the main dispatching device 111 can perform CRUSH calculation on the representative value HIDb of the data block ChunkB, and transmit the representative value HIDb and the data block ChunkB to the storage devices 120_4 and 120_5 (destination storage devices) according to the calculation result (hypothesis).

After the destination storage device answers (indicating successful storage), the main dispatching device 111 may provide the look-up table to the backup dispatching device 112 (step S670). In step S680, the main dispatching device 111 may reply to the user device 10 to indicate successful storage. The content of the look-up table of the backup dispatching device 112 is consistent with the content of the look-up table of the main dispatching device 111. Therefore, in the case that the lookup table of the main dispatching device 111 is damaged, or the main dispatching device 111 cannot operate normally, the backup dispatching device 112 can replace the main dispatching device 111 to perform the write operation flow shown in FIG. 6 And (or) the read operation flow shown in Figure 7.

FIG. 7 is a schematic diagram illustrating a read operation flow of the master dispatching device 111 shown in FIG. 5 according to an embodiment of the present invention. The functions of the backup dispatching device 112 shown in FIG. 5 can be deduced by analogy with reference to the relevant description of the main dispatching device 111. Please refer to Figure 5 and Figure 7. In step S710, the master dispatching device 111 may receive a data reading request from the user device 10. The data read request may include the original key value OID. In step S720, the main dispatching device 111 may search for the original key value OID corresponding to the data read request in the lookup table of the main dispatching device 111, so as to retrieve the representative value corresponding to the original key value OID from the lookup table. In step S730, the main dispatching device 111 may perform a distribution calculation on the representative value to determine the destination location corresponding to the representative value. In step S740, the main dispatching device 111 may transmit the read request and the representative value to one (or more) of the storage devices 120_1 to 120_n through the communication network 20 according to the destination location.

For example, the main dispatching device 111 may search for the original key value OID in the lookup table of the main dispatching device 111, so as to retrieve the representative values HIDa and HIDb corresponding to the original key value OID from the lookup table. The main dispatching device 111 may perform CRUSH calculation on the representative value HIDa, and transmit the read request and the representative value HIDa to the storage device 120_1 (destination storage device) according to the calculation result (hypothesis). When the storage device 120_2 is used as a backup device of the storage device 120_1, when the data of the storage device 120_1 fails, the master dispatching device 111 can transmit the read request and the representative value HIDa to the storage device 120_2 (destination storage device). In the same way, the main dispatching device 111 can perform CRUSH calculation on the representative value HIDb, and transmit the read request and the representative value HIDb to the storage device 120_4 (destination storage device) according to the calculation result (hypothesis). When the storage device 120_5 is used as a backup device of the storage device 120_4, when the data of the storage device 120_4 fails, the master dispatching device 111 may transmit the read request and the representative value HIDa to the storage device 120_5 (destination storage device).

The destination storage device can read the corresponding data block in its local storage space according to the representative value provided by the main dispatching device 111, and then send the data block back to the main dispatching device 111 according to the read request. After the master dispatching device 111 receives the data blocks returned by these destination storage devices (step S750), the master dispatching device 111 can aggregate these returned data blocks as the original data D, and return the original data D to the user device 10 (Step S760). For example, the storage device 120_1 (destination storage device) can read the corresponding data block ChunkA in its local storage space according to the representative value HIDa provided by the master dispatching device 111, and then transfer the data block according to the read request ChunkA is transmitted back to the main dispatching device 111. Similarly, the storage device 120_4 (destination storage device) can read the corresponding data block ChunkB in its local storage space according to the representative value HIDb provided by the main dispatching device 111, and then transfer the data block ChunkB according to the read request Return to the main dispatching device 111. After the main dispatching device 111 receives the data blocks ChunkA and ChunkB returned by the destination storage devices 120_1 and 120_4, the main dispatching device 111 can aggregate these data blocks ChunkA and ChunkB as the original data D, and combine the original key value OID with the original data D is sent back to the user device 10.

FIG. 8 is a circuit block diagram illustrating the dispatching device 110 shown in FIG. 1 or FIG. 2 according to another embodiment of the present invention. According to design requirements, the embodiment described in FIG. 8 can refer to the related descriptions of FIG. 3 and (or) FIG. 4. In the embodiment shown in FIG. 8, the dispatching device 110 includes an erasure coding device 113 and one (or more) data dispatching devices 114. The erasure coding device 113 and the data distribution device 114 are suitable for coupling to the communication network 20. The erasure coding device 113 and the data dispatching device 114 are two independent electronic devices. According to design requirements, any one of the erasure coding device 113 and the data distribution device 114 can be a personal computer, a workstation, a server, a NAS device, and/or other electronic devices. According to design requirements, in some embodiments, the erasure coding device 113 (or the data dispatching device 114) may be another electronic device independent of the storage devices 120_1 to 120_n. In other embodiments, at least one of the storage devices 120_1 to 120_n may be selected as the erasure coding device 113 (or the data dispatching device 114).

According to different design requirements, the implementation of the blocks of the erasure coding device 113 and/or the data dispatching device 114 can be hardware, firmware, software (ie, programs), or a combination of more of the three. form. In terms of software and/or firmware, the related functions of the erasure coding device 113 and/or the data dispatching device 114 described above can be implemented as programming codes. For example, a general programming language (such as C, C++, or assembly language) or other suitable programming languages are used to implement the erasure coding device 113 and/or the data dispatching device 114. The programming code can be recorded/stored in a recording medium. A computer, CPU, controller, microcontroller, or microprocessor can read and execute the programming code from the recording medium, so as to realize the related functions of the erasure coding device 113 and/or the data dispatching device 114. . Moreover, the programming code can also be provided to the computer (or CPU) via any transmission medium (communication network or broadcast wave, etc.).

FIG. 9 is a schematic diagram illustrating the writing operation flow of the erasure correction coding device 113 and the data dispatching device 114 shown in FIG. 8 according to an embodiment of the present invention. The function of the data dispatching device 114 shown in FIG. 8 can be deduced by referring to the related description of the erasure coding device 113. Please refer to Figure 8 and Figure 9. In step S900, the erasure coding device 113 may receive a data write request from the user device 10. The data write request may include the original key value OID and the original data D.

In step S905, the erasure correction coding device 113 may perform erasure correction coding calculations on the original data D to generate multiple erasure correction data and multiple key values corresponding to the erasure correction data. In step S910, the erasure coding device 113 may perform a distributed calculation on each of these key values to generate multiple device positions corresponding to these key values. In step S915, the erasure correction coding device 113 can transmit the erasure correction data and these key values to the multiple data distribution devices 114 via the communication network 20 according to the location of the device. For example, the erasure correction coding device 113 can perform erasure correction coding calculations on the original data D to generate a plurality of erasure correction data Shard1, Shard2, Shard3, Shard4, and Shard5. The erasure coding device 113 can also generate the key values OID1, OID2, OID3, OID4, and OID5 corresponding to the erasure data Shard1 to Shard5 according to the original key value OID. The erasure coding device 113 can perform distributed calculation on the key value OID1 to transmit the erasure correction data Shard1 and the key value OID1 to the first data distribution device of the data distribution devices 114 through the communication network 20. In the same way, the erasure coding device 113 can transmit the erasure data Shard2 and the key value OID2 to the second data distribution device of these data distribution devices 114 through the communication network 20, and transmit the erasure data Shard3 and the key value OID3 to these data distribution devices 114. The third data distribution device in the data distribution device 114 transmits the erasure data Shard4 and the key value OID4 to the fourth data distribution device in the data distribution devices 114, and the erasure data Shard5 and the key value OID5 are transmitted to The fifth data distribution device among these data distribution devices 114.

In step S950, these data allocating devices 114 can cut the erasure data into one (or more) data chunks. In step S955, these data allocating devices 114 may perform summary calculation on a current data block in this (or these) data blocks to generate a representative value corresponding to the current data block. In step S960, these data allocating devices 114 can record the mapping relationship between the original key value and the representative value in the look-up table of the data allocating device 114. For example, the first data allocating device among these data allocating devices 114 can cut the erasure data Shard1 into data blocks Chunk1A (or more data blocks), and the first data allocating device can compare data The block Chunk1A performs hash calculation to generate the representative value HID1a, and the first data distribution device can record the mapping relationship between the key value OID1 and the representative value HID1a in the lookup table of the first data distribution device. In the same way, the operations of the second to fifth data dispatching devices in these data dispatching devices 114 can be deduced by analogy with reference to the relevant description of the first data dispatching device in these data dispatching devices 114, so they will not be repeated here.

In step S965, these data allocating devices 114 may perform a distribution calculation on the representative value to determine the destination location corresponding to the representative value. In step S970, the data allocating devices 114 can transmit the current data block and the representative value to one (or more) of the storage devices 120_1 to 120_n through the communication network 20 according to the destination location. For example, the first data allocating device among the data allocating devices 114 may perform a CRUSH calculation on the representative value HID1a of the data block Chunk1A, and transmit the representative value HID1a and the data block Chunk1A to the storage device 120_1 according to the calculation result (hypothesis) (Destination storage device).

After the destination storage device answers (indicating successful storage), the data allocating device 114 can answer the erasure coding device 113 (step S975). After the data allocating device 114 answers (indicating successful storage), the erasure coding device 113 can answer (indicating successful storage) the user device 10.

FIG. 10 is a schematic diagram illustrating the read operation flow of the erasure correction coding device 113 and the data dispatching device 114 shown in FIG. 8 according to an embodiment of the present invention. Please refer to Figure 8 and Figure 10. In step S1000, the erasure coding device 113 may receive a data read request from the user device 10. The data read request may include the original key value OID. In step S1005, the erasure correction coding device 113 may perform erasure correction coding calculations on the original key value OID to generate multiple key values. In step S1010, the erasure coding device 113 may perform distributed calculation on each of these key values to generate multiple device positions corresponding to these key values. In step S1015, the erasure coding device 113 can transmit the read request and these key values to a plurality of data distribution devices 114 via the communication network 20 according to the location of the device.

For example, the erasure coding device 113 may perform erasure coding calculation on the original key value OID to generate multiple key values OID1, OID2, OID3, OID4, and OID5. The erasure coding device 113 can perform distributed calculation on the key value OID1, so as to transmit the read request and the key value OID1 to the first data dispatching device among the data dispatching devices 114 through the communication network 20. In the same way, the erasure coding device 113 can transmit the read request and the key value OID2 to the second data dispatching device of the data dispatching devices 114 through the communication network 20, and transmit the read request and the key value OID3 to the data dispatching device 114 The third data dispatching device in the data dispatching device 114 transmits the read request and the key value OID4 to the fourth data dispatching device of these data dispatching devices 114, and the read request and the key value OID5 are transmitted to the first data dispatching device 114 of these data dispatching devices 114 Five data distribution devices.

In step S1050, the data allocating device 114 may search for the key value corresponding to the read request in the look-up table of the data allocating device 114, so as to retrieve the representative value corresponding to the key value from the look-up table. In step S1055, the data allocating device 114 may perform a distribution calculation on the representative value to determine the destination location corresponding to the representative value. In step S1060, the data distribution device 114 may transmit the read request and the representative value to one (or more) of the storage devices 120_1 to 120_n through the communication network 20 according to the destination location.

For example, the first data allocating device among the data allocating devices 114 may search for the key value OID1 in the lookup table of the first data allocating device, so as to retrieve the representative corresponding to the key value OID1 from the lookup table. Value HID1a. The first data distribution device may perform CRUSH calculation on the representative value HID1a, and transmit the read request and the representative value HID1a to the storage device 120_1 (destination storage device) according to the calculation result (hypothesis). In the same way, the operations of the second to fifth data dispatching devices in these data dispatching devices 114 can be deduced by analogy with reference to the relevant description of the first data dispatching device in these data dispatching devices 114, so they will not be repeated here.

These destination storage devices can read corresponding data blocks in their local storage space according to the representative values provided by the data allocating devices 114, and then send the data blocks back to the data allocating devices 114 according to the read request. After the data allocating devices 114 receive the data blocks returned by the destination storage devices (step S1065), the data allocating devices 114 can aggregate the returned data blocks as erasure data, and return the erasure data to erasure correction Encoding device 113 (step S1070). After the erasure-correction coding device 113 receives the erasure-corrected data returned by the data dispatching device 114, the erasure-correction coding device 113 can aggregate the returned erasure-corrected data as the original data D, and return the original key value OID and the original data D. It is transmitted to the user device 10 (step S1020).

For example, the storage device 120_1 (destination storage device) can read the corresponding data block in its local storage space according to the representative value HID1a provided by the first data distribution device among these data distribution devices 114 Chunk1A, and then send the data chunk Chunk1A back to the first data distribution device according to the read request. After the first data distribution device receives the data block Chunk1A returned by the destination storage device 120_1, the first data distribution device can aggregate this data block ChunkA as the erasure data Shard1, and combine the key value OID1 with the erasure data. The deleted data Shard1 is sent back to the erasure coding device 113. In the same way, the operations of the second to fifth data dispatching devices in these data dispatching devices 114 can be deduced by analogy with reference to the relevant description of the first data dispatching device in these data dispatching devices 114, so they will not be repeated here. After the erasure coding device 113 receives the erasure correction data Shard1 to Shard5 returned by the data dispatching device 114, the erasure coding device 113 can aggregate the erasure correction data Shard1 to Shard5 as the original data D, and combine the original key value OID with The original data D is returned to the user device 10.

FIG. 11 is a circuit block diagram illustrating the dispatching device 110 shown in FIG. 1 or FIG. 2 according to another embodiment of the present invention. According to design requirements, the embodiment described in FIG. 11 can refer to the related descriptions of FIG. 3 and (or) FIG. 4. In the embodiment shown in FIG. 11, the dispatching device 110 includes an erasure correction coding device 115 and a backup erasure correction coding device 116. The erasure correction coding device 115 and the redundant erasure correction coding device 116 are adapted to be coupled to the communication network 20. The erasure correction coding device 115 and the backup erasure correction coding device 116 are two independent electronic devices. According to design requirements, any one of the erasure correction coding device 115 and the backup erasure coding device 116 may be a personal computer, a workstation, a server, a NAS device, and/or other electronic devices. According to design requirements, in some embodiments, the erasure correction coding device 115 (or the backup erasure correction coding device 116) may be another electronic device independent of the storage devices 120_1 to 120_n. In other embodiments, at least one of these storage devices 120_1 to 120_n may be selected as the erasure correction coding device 115 (or backup erasure coding device 116).

According to different design requirements, the implementation of the blocks of the erasure coding device 115 and/or the backup erasure coding device 116 can be hardware, firmware, software (ie, programs) or more of the foregoing three. The combination form of the person. In the form of software and/or firmware, the related functions of the erasure correction coding device 115 and/or the backup erasure coding device 116 can be implemented as programming codes. For example, a general programming language (for example, C, C++, or assembly language) or other suitable programming languages are used to implement the erasure correction coding device 115 and/or the backup erasure coding device 116. The programming code can be recorded/stored in a recording medium. A computer, CPU, controller, microcontroller, or microprocessor can read and execute the programming code from the recording medium, thereby realizing the erasure correction coding device 115 and/or the backup erasure coding device 116. Related functions. Moreover, the programming code can also be provided to the computer (or CPU) via any transmission medium (communication network or broadcast wave, etc.).

FIG. 12 is a schematic diagram illustrating the writing operation flow of the erasure correction coding device 115 shown in FIG. 11 according to an embodiment of the present invention. The function of the redundant erasure correction coding device 116 shown in FIG. 11 can be deduced by analogy with reference to the related description of the erasure correction coding device 115. Please refer to Figure 11 and Figure 12. In step S1210, the erasure coding device 115 may receive a data write request from the user device 10. The data write request may include the original key value OID and the original data D.

In step S1220, the erasure correction coding device 115 may perform erasure correction coding calculations on the original data D to generate multiple erasure correction data and multiple key values corresponding to the erasure correction data. For example, the erasure correction coding device 115 may perform erasure correction coding calculations on the original data D to generate a plurality of erasure correction data Shard1, Shard2, Shard3, Shard4, and Shard5. The erasure coding device 115 can also generate the key values OID1, OID2, OID3, OID4, and OID5 corresponding to the erasure data Shard1 to Shard5 according to the original key value OID.

In step S1230, the erasure correction coding device 115 may cut each of these erasure correction data into one (or more) data chunks. In step S1240, the erasure coding device 115 may perform digest calculation on a current data block in this (or these) data blocks to generate a representative value corresponding to the current data block. In step S1250, the erasure correction coding device 115 may record the mapping relationship between the key value and the representative value in the lookup table of the erasure correction coding device 115. For example, the erasure correction coding device 115 can cut the erasure correction data Shard1 into data blocks Chunk1A (or more data blocks). Similarly, the erasure coding device 115 can cut the erasure data Shard2 into data blocks Chunk2A, the erasure data Shard3 into data blocks Chunk3A, the erasure data Shard4 into data blocks Chunk4A, and the erasure data Shard5 into Data block Chunk5A. The erasure coding device 115 can perform a hash calculation on the data block Chunk1A to generate a representative value HID1a, perform a hash calculation on the data block Chunk2A to generate a representative value HID2a, and perform a hash calculation on the data block Chunk3A to generate a representative value HID3a. The block Chunk4A is hashed to generate the representative value HID4a, and the data block Chunk5A is hashed to generate the representative value HID5a. The erasure coding device 115 may record the mapping relationship between the key values OID1 to OID5 and the representative values HID1a to HID5a in the lookup table of the erasure coding device 115.

In step S1260, the erasure correction coding device 115 may perform a distribution calculation on the representative value to determine the destination position corresponding to the representative value. In step S1270, the erasure coding device 115 can transmit the current data block and the representative value to the destination storage device among the storage devices 120_1 to 120_n through the communication network 20 according to the destination location. For example, the erasure coding device 115 may perform a CRUSH calculation on the representative value HID1a of the data block Chunk1A, and transmit the representative value HID1a and the data block Chunk1A to the storage device 120_1 (destination storage device) according to the calculation result (hypothesis). Similarly, the erasure coding device 115 can perform CRUSH calculations on the representative values HID2a to HID5a of the data blocks Chunk2A to Chunk5A, and transmit the representative values HID2a to HID5a and the data blocks Chunk2A to Chunk5A to the storage device 120_2 according to the calculation result (hypothesis). ~120_5 (Destination storage device).

After the destination storage device answers (indicating successful storage), the erasure correction coding device 115 may provide the look-up table to the backup erasure coding device 116 (step S1280). In step S1290, the erasure coding device 115 may reply to the user device 10 to indicate successful storage. The content of the lookup table of the backup erasure correction coding device 116 is consistent with the content of the lookup table of the erasure correction coding device 115. Therefore, in the case where the look-up table of the erasure correction coding device 115 is damaged, or in the case where the erasure correction coding device 115 cannot operate normally, the backup erasure coding device 116 can replace the erasure correction coding device 115 to perform FIG. 12 The write operation flow shown and/or the read operation flow shown in FIG. 13.

FIG. 13 is a schematic diagram illustrating the read operation flow of the erasure correction coding device 115 shown in FIG. 11 according to an embodiment of the present invention. The function of the redundant erasure correction coding device 116 shown in FIG. 11 can be deduced by analogy with reference to the related description of the erasure correction coding device 115. Please refer to Figure 11 and Figure 13. In step S1310, the erasure coding device 115 may receive a data read request from the user device 10. The data read request may include the original key value OID. In step S1320, the erasure coding device 115 may perform erasure coding calculation on the original key value OID corresponding to the data read request to generate multiple key values. In step S1330, the erasure correction coding device 115 may search for these key values in the lookup table of the erasure correction coding device 115, so as to retrieve the representative values corresponding to these key values from the lookup table. In step S1340, the erasure correction coding device 115 may perform a distribution calculation on the representative values to determine the destination positions corresponding to the representative values. In step S1350, the erasure coding device 115 can transmit the read request and the representative values to one (or more) of the storage devices 120_1 to 120_n through the communication network 20 according to the destination location.

For example, the erasure coding device 115 may perform erasure coding calculation on the original key value OID to generate multiple key values OID1, OID2, OID3, OID4, and OID5. The erasure coding device 115 can search for the key value OID1 in the lookup table of the erasure coding device 115, so as to retrieve the representative value HID1a corresponding to the key value OID1 from the lookup table. Similarly, the erasure coding device 115 can search for the key values OID2 to OID5 in the lookup table, so as to retrieve the representative values HID2a to HID5a corresponding to the key values OID2 to OID5 from the lookup table. The erasure coding device 115 may perform CRUSH calculation on the representative values HID1a to HID5a, and according to the calculation result (hypothesis), transmit the read request and the representative values HID1a to HID5a to the storage devices 120_1 to 120_5 (destination storage devices), respectively.

The destination storage device can read the corresponding data block in its local storage space according to the representative value provided by the erasure coding device 115, and then send the data block back to the erasure coding device 115 according to the read request . After the erasure coding device 115 receives the data blocks returned by these destination storage devices (step S1360), the erasure coding device 115 may aggregate these returned data blocks as erasure data (step S1370). The erasure coding device 115 can aggregate these erasure data as the original data D, and return the original key value OID and the original data D to the user device 10 (step S1380).

For example, the storage devices 120_1 to 120_5 (destination storage devices) can read the corresponding data blocks Chunk1A to Chunk5A in their local storage space according to the representative values HID1a to HID5a provided by the erasure coding device 115, and then follow The read request returns these data blocks Chunk1A to Chunk5A to the erasure coding device 115. After the erasure coding device 115 receives the data blocks Chunk1A to Chunk5A returned by the destination storage devices 120_1 to 120_5, the erasure coding device 115 can aggregate these data blocks Chunk1A to Chunk5A as erasure data Shard1 to Shard5. The erasure coding device 115 can aggregate the erasure data Shard1 to Shard5 as the original data D, and return the original key value OID and the original data D to the user device 10.

In summary, the data storage system and its global deduplication method described in the foregoing embodiments can be applied to a distributed storage system. The allocating device 110 can generate the representative value HID corresponding to the current data block, and determine the destination location according to the representative value HID. According to the destination location, the allocating device 110 can transmit the current data block and the representative value HID to at least one destination storage device among the multiple storage devices via the communication network 20. In other words, the same data block will be sent to the same storage device. The destination storage device can check the representative value HID to determine whether to store the current data block in its local storage space. Therefore, the data storage system can effectively de-duplicate in a distributed storage environment.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to those defined by the attached patent application scope.

10: User device 20, 30: Communication network 100, 200: data storage system 110: Dispatch device 111: Main Dispatch Device 112: Redundant Dispatching Device 113, 115: erasure coding device 114: Data Distribution Device 116: Redundant erasure coding device 120_1, 120_2, 120_3, 120_4, 120_5, 120_n: storage device S310～S360, S410～S490, S610～S680, S710～S760, S900～S975, S1000～S1070, S1210～S1290, S1310～S1380: steps

FIG. 1 is a schematic diagram of a circuit block of a data storage system according to an embodiment of the present invention. FIG. 2 is a schematic block diagram of a circuit of a data storage system according to another embodiment of the present invention. Fig. 3 is a schematic flowchart of a global de-duplication method according to an embodiment of the present invention. FIG. 4 is a schematic diagram of the operation flow of a storage device according to an embodiment of the present invention. FIG. 5 is a circuit block diagram illustrating the dispatching device shown in FIG. 1 or FIG. 2 according to an embodiment of the present invention. FIG. 6 is a schematic diagram illustrating a write operation flow of the master dispatching device shown in FIG. 5 according to an embodiment of the present invention. FIG. 7 is a schematic diagram illustrating a read operation flow of the master dispatching device shown in FIG. 5 according to an embodiment of the present invention. FIG. 8 is a circuit block diagram illustrating the dispatching device shown in FIG. 1 or FIG. 2 according to another embodiment of the present invention. 9 is a schematic diagram illustrating the writing operation flow of the erasure correction coding device and the data dispatching device shown in FIG. 8 according to an embodiment of the present invention. FIG. 10 is a schematic diagram illustrating the read operation flow of the erasure correction coding device and the data dispatching device shown in FIG. 8 according to an embodiment of the present invention. FIG. 11 is a circuit block diagram illustrating the dispatching device shown in FIG. 1 or FIG. 2 according to another embodiment of the present invention. FIG. 12 is a schematic diagram illustrating a write operation flow of the erasure correction coding apparatus shown in FIG. 11 according to an embodiment of the present invention. FIG. 13 is a schematic diagram illustrating a read operation flow of the erasure correction coding apparatus shown in FIG. 11 according to an embodiment of the present invention.

S310~S360: steps

Claims (26)

A data storage system including: A plurality of storage devices, suitable for being coupled to a communication network; and A dispatching device adapted to receive a data write request, wherein the dispatching device is configured to cut an original data corresponding to the data write request into at least one data block, and the dispatching device responds to one of the at least one data block The current data block performs a summary calculation to generate a representative value corresponding to the current data block, the distribution device performs a first distribution calculation on the representative value to determine a destination location corresponding to the representative value, and the distribution device according to The destination location transmits the current data block and the representative value to at least one destination storage device among the storage devices through the communication network; The at least one destination storage device checks the representative value to determine whether to store the current data block in a storage space of the at least one destination storage device. The data storage system according to claim 1, wherein at least one of the storage devices is selected as the allocating device. The data storage system according to claim 1, wherein the digest calculation includes a hash algorithm, and the representative value includes a hash value of the current data block. The data storage system according to claim 1, wherein the first distributed calculation includes a Ceph CRUSH algorithm. The data storage system according to claim 1, wherein the at least one destination storage device checks a lookup table for the representative value, When the lookup table has the representative value, the at least one destination storage device gives up storing the current data block in the storage space, increases a reference count value corresponding to the representative value, and updates the reference count value to The lookup table; and When the lookup table does not have the representative value, the at least one destination storage device stores the current data block in a physical address of the storage space, sets the reference count value corresponding to the representative value to an initial value, and The representative value, the physical address, and the reference count value are recorded in the look-up table. The data storage system according to claim 1, wherein the data write request further includes an original key value, and the dispatching device records a mapping relationship between the original key value and the representative value in a lookup table. The data storage system according to claim 1, wherein the allocation device includes a main allocation device and a backup allocation device, the data write request further includes an original key value, and the main allocation device associates the original key value with the representative A mapping relationship of the value is recorded in a look-up table, and the look-up table is provided to the spare allocation device. The data storage system according to claim 1, wherein the at least one destination storage device includes a first destination storage device and a second destination storage device, and the allocating device transmits the current data block and the representative value through the communication network To the first destination storage device and the second destination storage device. The data storage system according to claim 1, wherein the allocation device includes an erasure coding device and at least one data allocation device, the erasure coding device receives the data write request, and the erasure coding device performs an erasure correction on the original data. Erasure correction coding is calculated to generate at least one erasure correction data and at least one key value corresponding to the at least one erasure correction data, and the erasure correction coding device performs a second distribution calculation on the at least one key value to generate the at least one key value Corresponding to at least one device location, the erasure coding device transmits the at least one erasure correction data and the at least one key value to the at least one data distribution device through the communication network according to the at least one device location, the at least one data The allocation device cuts the at least one erasure data into the at least one data block, the at least one data allocation device performs the summary calculation on the current data block in the at least one data block to generate the representative value, the at least one data The distribution device performs the first distribution calculation on the representative value to determine the destination location, and the at least one data distribution device transmits the current data block and the representative value to the at least one destination storage via the communication network according to the destination location Device. The data storage system according to claim 9, wherein the at least one data distribution device records a mapping relationship between the at least one key value and the representative value in a look-up table. The data storage system according to claim 1, wherein the allocating device includes an erasure coding device, the erasure coding device receives the data write request, and the erasure coding device performs an erasure coding calculation on the original data to generate At least one erasure correction data and at least one key value corresponding to the at least one erasure correction data, the erasure correction coding device cuts the at least one erasure correction data into the at least one data block, and the erasure correction coding device pairs the at least one erasure correction data in the at least one data block. The current data block in the data block performs the digest calculation to generate the representative value, the erasure coding device performs the first distribution calculation on the representative value to determine the target position, and the erasure coding device performs the calculation according to the target position The current data block and the representative value are transmitted to the at least one destination storage device through the communication network. The data storage system according to claim 11, wherein the erasure coding device records a mapping relationship between the at least one key value and the representative value in a look-up table. The data storage system according to claim 12, wherein the allocating device further includes a redundant erasure correction coding device, and the erasure correction coding device provides the look-up table to the redundant erasure coding device. A global deduplication method, including: Receive a data write request by a dispatch device; The dispatching device cuts an original data corresponding to the data write request into at least one data block; Performing a summary calculation on a current data block in the at least one data block by the allocating device to generate a representative value corresponding to the current data block; The allocating device performs a first distribution calculation on the representative value to determine a destination location corresponding to the representative value; The allocating device transmits the current data block and the representative value to at least one destination storage device among a plurality of storage devices through a communication network according to the destination location; and The at least one destination storage device checks the representative value to determine whether to store the current data block in a storage space of the at least one destination storage device. The global deduplication method according to claim 14, wherein at least one of the storage devices is selected as the allocating device. The global deduplication method according to claim 14, wherein the digest calculation includes a hash algorithm, and the representative value includes a hash value of the current data block. The global deduplication method according to claim 14, wherein the first distributed calculation includes a Ceph CRUSH algorithm. The global deduplication method described in claim 14 further includes: Check whether a lookup table has the representative value by the at least one destination storage device; When the lookup table has the representative value, give up storing the current data block in the storage space, increase a reference count value corresponding to the representative value, and update the reference count value to the lookup table; and When the look-up table does not have the representative value, store the current data block in a physical address of the storage space, set the reference count value corresponding to the representative value to an initial value, and set the representative value, the representative value, and the The physical address and the reference count value are recorded in the look-up table. The global deduplication method according to claim 14, wherein the data write request further includes an original key value, and the global deduplication method further includes: The dispatching device records a mapping relationship between the original key value and the representative value in a lookup table. The global deduplication method according to claim 14, wherein the dispatch device includes a primary dispatch device and a backup dispatch device, the data write request further includes an original key value, and the global deduplication method further includes: The master dispatching device records a mapping relationship between the original key value and the representative value in a lookup table; and The main dispatch device provides the look-up table to the spare dispatch device. The global deduplication method according to claim 14, wherein the at least one destination storage device includes a first destination storage device and a second destination storage device, and the global deduplication method further includes: The distribution device transmits the current data block and the representative value to the first destination storage device and the second destination storage device through the communication network. The global deduplication method according to claim 14, wherein the dispatching device includes an erasure correction coding device and at least one data dispatching device, and the global deduplication method further includes: Receiving the data write request by the erasure coding device; Performing an erasure correction coding calculation on the original data by the erasure correction coding device to generate at least one erasure correction data and at least one key value corresponding to the at least one erasure correction data; Performing a second distribution calculation on the at least one key value by the erasure coding device to generate at least one device position corresponding to the at least one key value; The erasure coding device transmits the at least one erasure correction data and the at least one key value to the at least one data distribution device through the communication network according to the location of the at least one device; Cutting the at least one erasure-corrected data into the at least one data block by the at least one data allocating device; Performing the summary calculation on the current data block in the at least one data block by the at least one data allocating device to generate the representative value; The at least one data distribution device performs the first distribution calculation on the representative value to determine the destination location; and The at least one data distribution device transmits the current data block and the representative value to the at least one destination storage device through the communication network according to the destination location. The global deduplication method described in claim 22 further includes: The at least one data allocating device records a mapping relationship between the at least one key value and the representative value in a look-up table. The global deduplication method according to claim 14, wherein the dispatching device includes an erasure correction coding device, and the global deduplication method further includes: Receiving the data write request by the erasure coding device; Performing an erasure correction coding calculation on the original data by the erasure correction coding device to generate at least one erasure correction data and at least one key value corresponding to the at least one erasure correction data; Cutting the at least one erasure correction data into the at least one data block by the erasure correction coding device; Performing the digest calculation on the current data block in the at least one data block by the erasure coding device to generate the representative value; The erasure coding device performs the first distribution calculation on the representative value to determine the target location; and The erasure coding device transmits the current data block and the representative value to the at least one destination storage device through the communication network according to the destination location. The global deduplication method described in claim 24 further includes: The erasure coding device records a mapping relationship between the at least one key value and the representative value in a look-up table. The global deduplication method according to claim 25, wherein the dispatching device further includes a redundant erasure coding device, and the global deduplication method further includes: The look-up table is provided by the erasure correction coding device to the backup erasure coding device.

Images