WO2023040200A1

WO2023040200A1 - Data deduplication method and system, and storage medium and device

Info

Publication number: WO2023040200A1
Application number: PCT/CN2022/078324
Authority: WO
Inventors: 刚亚州
Original assignee: 苏州浪潮智能科技有限公司
Priority date: 2021-09-17
Filing date: 2022-02-28
Publication date: 2023-03-23
Also published as: CN113535708A

Abstract

Provided in the present application are a data deduplication method and system, and a storage medium and a device. The method comprises: in response to a host abandoning a data deduplication operation and writing, into a hard disk, data taking granularity as a unit, calculating a fingerprint value of the data, writing the fingerprint value and a logical address thereof into the hard disk, and determining whether a storage space occupied in the hard disk reaches a preset threshold value; if the preset threshold value is reached, acquiring unit data from the hard disk, and determining whether a deduplication operation has been performed on the unit data; if the deduplication operation has not been performed, acquiring the fingerprint value of the unit data, and querying, by means of a metadata management module, whether there is original metadata which includes an original physical address which has a mapping relationship with the fingerprint value of the unit data; and if so, establishing first metadata which includes the mapping relationship between the original physical address and the fingerprint value of the unit data, and storing the first metadata in the metadata management module for the deduplication operation performed on the unit data.

Description

A data deduplication method, system, storage medium and equipment

Cross References to Related Applications

This application claims the priority of the Chinese patent application submitted to the China Patent Office on September 17, 2021, with the application number 202111090326.4, and the application name is "a data deduplication method, system, storage medium and equipment", and the entire content of the application is passed References are incorporated in this application.

technical field

The present application relates to a data deduplication method, system, storage medium and equipment.

Background technique

Metadata (Mete data) refers to the data describing data (data about data), which can be understood as data with a wider range of data than the general meaning, not only indicating the type, name, value and other information of the data, but also further providing The context information of the data, such as the domain to which the data belongs, the source of the data, and so on. In a data storage system, metadata is the basis of information storage and the smallest unit of data. In recent years, with the development of information technology, massive data has been generated, but how to effectively manage and organize these massive data has become a prominent problem. For a large amount of stored data, querying and analyzing the data content and meaning of the data can make more effective use of the data. In the storage system, the efficient organization and management of metadata is an effective means to solve this problem, and can support the management and maintenance of data by the system. Therefore, data becomes more valuable only when metadata is managed effectively.

All-flash storage is a storage system based on all-flash arrays. It is an independent storage array or device completely composed of solid-state storage media. The main difference from traditional hard disk storage is higher performance, faster and more stable data processing. In the all-flash storage system, the data online deduplication feature is the most important and necessary feature, because the back-end of the all-flash storage system uses solid-state drives as storage media. In view of the value of solid-state drives, all-flash storage systems require data to be online. Deduplication is used to reduce the actual storage space of the backend disk. To realize the online deduplication of the all-flash storage system, metadata management is very important. Metadata management mainly manages L-P (LBA→PBA) mapping, P-L (PBA→LBA) mapping, and H-P (HASHKEY→PBA) mapping relationship. Among them, LBA (Logical Block Address) represents the logical block address, PBA (Physical Block Address) represents the physical block address, and HASHKEY represents the hash value. Compared with the traditional feature that does not support online deduplication, metadata management has more metadata of P-L mapping and H-P mapping relationship, which involves a large amount of data access with high concurrency and short delay, which puts more pressure on metadata management. big.

In some special scenarios, such as a controller failure or high data writing pressure so that the performance cannot meet the requirements, some online deduplication requests will be abandoned to meet the performance. However, the inventor realizes that in this way, some data that should be deduplicated will not be deduplicated, thereby occupying more storage space of the back-end solid-state disk.

Contents of the invention

The embodiment of the present application provides a data deduplication method, including the following steps:

In response to the host giving up the deduplication operation and writing the data in the unit of granularity to the hard disk, calculating the fingerprint value of the data, and writing the fingerprint value of the data and its logical address to the hard disk, and judging that the hard disk is occupied Whether the storage space of the user reaches the preset threshold;

In response to the occupied storage space reaching a preset threshold, acquire the unit data from the hard disk, and determine whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data;

Responding to the fact that the unit data has not been deduplicated, obtain the fingerprint value of the unit data, and query whether there is original metadata including the original physical address that has a mapping relationship with the fingerprint value of the unit data through the metadata management module; and

In response to the existence of the original metadata, first metadata including the mapping relationship between the original physical address and the fingerprint value of the unit data is established, and the first metadata is stored in the metadata management module for deduplication of the unit data.

In one or more embodiments, judging whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data includes:

Determine whether the unit data has a corresponding fingerprint value and logical address;

Confirming that the unit data has not been deduplicated in response to the unit data having a corresponding fingerprint value and logical address; and

In response to the fact that the unit of data has a corresponding fingerprint value but does not have a corresponding logical address, it is confirmed that the unit of data has undergone a deduplication operation.

In one or more embodiments, the method also includes:

In response to the existence of the original metadata, a second metadata group including the mapping relationship between the original physical address and the logical address of the unit data is established, and the second metadata group is stored in the metadata management module.

In one or more embodiments, the method also includes:

In response to storing the second metadata group into the metadata management module, invalidate the logical address of the unit data.

In one or more embodiments, the method also includes:

In response to storing both the first metadata and the second metadata group in the metadata management module, the garbage collection module is notified to perform garbage collection on the unit of data.

In one or more embodiments, establishing the second metadata group including the mapping relationship between the original physical address and the logical address of the unit data includes:

Metadata containing key-value pairs pointing to the logical address of the unit data from the original physical address and metadata containing key-value pairs pointing to the original physical address from the logical address of the unit data are established.

In one or more embodiments, the method also includes:

In response to the absence of original metadata, establishing third metadata including a mapping relationship between the fingerprint value of the unit data and its physical address, and establishing fourth metadata including a mapping relationship between the logical address of the unit data and its physical address, and The third metadata and the fourth metadata are stored in the metadata management module.

Another aspect of the present application also provides a data deduplication system, including:

The storage space judging module is configured to respond to the host giving up the deduplication data operation and write the data in the unit of granularity to the hard disk, calculate the fingerprint value of the data, and write the fingerprint value and its logical address of the data into the hard disk, and Determine whether the occupied storage space in the hard disk reaches a preset threshold;

The deduplication judging module is configured to obtain the unit data from the hard disk in response to the occupied storage space reaching a preset threshold, and judge whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data;

The original metadata query module is configured to obtain the fingerprint value of the unit data in response to the fact that the unit data has not been deduplicated, and query whether there is an original physical address that has a mapping relationship with the fingerprint value of the unit data through the metadata management module. raw metadata; and

The data deduplication module is configured to, in response to the existence of the original metadata, establish the first metadata including the mapping relationship between the original physical address and the fingerprint value of the unit data, and store the first metadata in the metadata management module for unit data Data deduplication operation.

In yet another aspect of the present application, one or more non-volatile computer-readable storage media storing computer-readable instructions is also provided. When the computer-readable instructions are executed by one or more processors, one or more The processor executes the steps of the data deduplication method in any of the foregoing embodiments.

In yet another aspect of the present application, a computer device is provided, including a memory and one or more processors, where computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the one or more processors, a or multiple processors execute the steps of the data deduplication method in any of the foregoing embodiments.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features and advantages of the application will be apparent from the description, drawings, and claims.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings that need to be used in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. Those of ordinary skill in the art can also obtain other embodiments according to these drawings without making creative efforts.

FIG. 1 is a schematic diagram of a data deduplication method provided according to one or more embodiments of the present application;

FIG. 2 is a schematic diagram of a data deduplication system provided according to one or more embodiments of the present application;

FIG. 3 is a schematic diagram of a non-volatile computer-readable storage medium for implementing a data deduplication method provided according to one or more embodiments of the present application;

Fig. 4 is a schematic diagram of a hardware structure of a computer device for performing a data deduplication method according to one or more embodiments of the present application.

Detailed ways

In order to make the technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

It should be noted that all the expressions using "first" and "second" in the embodiments of this application are to distinguish two entities with the same name or different parameters. It can be seen that "first" and "second" " is only for the convenience of expression, and should not be understood as limiting the embodiment of the present application. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, of a process, method, system, product or other steps or elements inherent in a process, method, system, product, or device comprising a series of steps or elements.

Based on the above purpose, the first aspect of the embodiments of the present application proposes an embodiment of a data deduplication method. FIG. 1 is a schematic diagram of a data deduplication method provided by one or more embodiments of the present application. The application of the method to computer equipment is taken as an example for illustration. As shown in Figure 1, the embodiment of this application may include the following steps:

Step S10, in response to the host giving up the operation of deduplicating data and writing the data in the unit of granularity to the hard disk, calculating the fingerprint value of the data, and writing the fingerprint value of the data and its logical address into the hard disk, and judging the hard disk Whether the occupied storage space reaches the preset threshold;

Step S20, in response to the occupied storage space reaching a preset threshold, acquiring the unit data from the hard disk, and judging whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data;

Step S30: Obtain the fingerprint value of the unit data in response to the fact that the unit data has not been deduplicated, and query whether there is original metadata including the original physical address that has a mapping relationship with the fingerprint value of the unit data through the metadata management module; and

Step S40, in response to the existence of the original metadata, establish the first metadata including the mapping relationship between the original physical address and the fingerprint value of the unit data, and store the first metadata in the metadata management module to perform the deduplication operation of the unit data .

In the embodiment of the present application, the deduplication operation does not mean deduplicating data as it is literally displayed. The deduplication operation refers to querying the HP (HASHKEY→PBA) mapping for the data newly written to the hard disk. (HASHKEY, fingerprint value) has a corresponding HP mapping, indicating that the data to which the H belongs already has data with the same content in the storage pool, so no physical address (PBA) will be assigned to it, so as to avoid data with the same content is called repeatedly. It should be noted that a piece of data that has not undergone a deduplication operation not only corresponds to a logical address (LBA) of a storage volume, but also has a physical address (PBA) corresponding to a storage pool. The fingerprint value is the hash value, which is mainly used as the unique identifier of the data content. If the content of two data is the same, the fingerprint value of the two is also the same, but the logical address and physical address of the two are not necessarily the same.

In the embodiment of the present application, the granularity (Grain) represents the minimum capacity unit of data, and data is written according to the granularity unit. The unit data in the embodiment of the present application means the data of one granularity unit.

In this embodiment of the application, the metadata is stored in the metadata management module. When traversing the hard disk and obtaining a unit of data in the hard disk, if the unit of data has not been deduplicated, the fingerprint value H of the unit of data is obtained, and the metadata management module is used to query whether it contains the fingerprint value H The original metadata of the original physical address P0 of the mapping relationship, if such original metadata exists, establish the first metadata including the mapping relationship between the original physical address P0 and the fingerprint value H of the unit data, and store the first metadata to the metadata management module. Assuming that the fingerprint value in the original metadata is H0, and this unit of data has its own physical address P, then H and H0 are the same, which means that the data content of the two is the same. Therefore, in order to achieve deduplication, H and P0 forms a mapping relationship, and stores the first metadata containing the mapping relationship into the metadata management module. At this time, the original metadata in the metadata management module is the same as the first metadata, and it is possible to avoid the metadata of the same content through the metadata. Data is called repeatedly.

The data deduplication method in the embodiment of the present application obtains the fingerprint value of the unit data that has not been deduplicated, and when the original metadata containing the mapping relationship between the fingerprint value of the unit data and the corresponding original physical address is found, Establish the first metadata including the mapping relationship between the original physical address and the fingerprint value of the unit data, and store the first metadata in the metadata management module, which avoids the impact on the performance of the storage system while realizing online deduplication. In this way, the requirements of the overall deduplication rate of the storage system are met, which is efficient and accurate; and the concurrency of access is improved by setting the metadata management module to obtain efficient metadata access.

In one or more embodiments, judging whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data includes: judging whether the unit data has a corresponding fingerprint value and logical address; If the unit data has a corresponding fingerprint value and no corresponding logical address, it is confirmed that the unit data has been deduplicated.

In the above-mentioned embodiment, if a unit of data is sent from the host through the traditional deduplication process, only its fingerprint value is stored in the hard disk, and its logical address is not stored.

In one or more embodiments, the method further includes: in response to the existence of the original metadata, establishing a second metadata group including the mapping relationship between the original physical address and the logical address of the unit data, and storing the second metadata group in Metadata management module.

In the above embodiments, in order to ensure the effectiveness of the management function of the metadata management module and the integrity of the metadata in the metadata management module, all the metadata related to the unit of data are stored in the metadata management module.

In one or more embodiments, the method further includes: invalidating the logical address of the unit of data in response to storing the second metadata group into the metadata management module.

In the above embodiment, by invalidating the logical address of the unit of data, it is also helpful to judge whether the unit of data has been deduplicated through the logical address.

In one or more embodiments, the method further includes: in response to storing both the first metadata and the second metadata group in the metadata management module, informing the garbage collection module to perform garbage collection on the unit of data.

In the above-mentioned embodiment, in order to further avoid data with repeated content from occupying memory space, a garbage collection mechanism is activated for the unit of data.

In one or more embodiments, establishing the second metadata group including the mapping relationship between the original physical address and the logical address of the unit data includes: establishing metadata including key-value pairs pointing to the logical address of the unit data from the original physical address , and metadata containing key-value pairs pointing from the logical address of the unit data to the original physical address.

In the above embodiment, in order to ensure the integrity of the metadata in the metadata management module, two metadata of two key-value pairs including the mapping relationship between the original physical address and the logical address of the unit data are stored in the metadata management module.

In one or more embodiments, the method further includes: in response to the absence of original metadata, establishing third metadata including the mapping relationship between the fingerprint value of the unit data and its physical address, and establishing the logical address and The fourth metadata of the mapping relation of the physical address, and store the third metadata and the fourth metadata in the metadata management module.

In the above embodiment, if there is no original metadata containing the mapping relationship between the fingerprint value of the unit data and the corresponding original physical address, it means that the unit data is new data, and there is no data with the same content, and naturally there will be no exceptions. Duplicate operation. Therefore, the corresponding mapping relationship is established for the fingerprint value, physical address, and logical address of the unit data itself, and the third metadata and fourth metadata with the corresponding mapping relationship are stored in the metadata management module, so that other When there is new data, it can be known whether there is a fingerprint value consistent with the data content of the unit by querying the metadata management module. The fourth metadata in this embodiment includes the metadata of the key-value pair pointing to the physical address from the logical address of the unit data, and also includes the metadata of the key-value pair pointing to the logical address from the physical address of the unit.

It should be understood that although the various steps in the flow chart of FIG. 1 are displayed sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Fig. 1 may include multiple sub-steps or multiple stages, these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, the execution of these sub-steps or stages The order is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

According to the second aspect of the embodiments of the present application, a data deduplication system is also provided. Fig. 2 is a schematic diagram of a data deduplication system provided according to one or more embodiments of the present application. As shown in Figure 2, a data deduplication system includes: a storage space judging module 10, configured to respond to the host giving up the deduplication data operation and write the data in granularity to the hard disk, calculate the fingerprint value of the data, And write the fingerprint value of the data and its logical address into the hard disk, and judge whether the occupied storage space in the hard disk reaches the preset threshold; the deduplication judging module 20 is configured to respond to the occupied storage space reaching the preset threshold, Acquire the unit data from the hard disk, and judge whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data; the original metadata query module 30 is configured to acquire the unit in response to the unit data not performing a deduplication operation The fingerprint value of the data, and whether there is original metadata containing the original physical address that has a mapping relationship with the fingerprint value of the unit data through the metadata management module; and the data deduplication module 40, configured to respond to the existence of the original metadata, The first metadata including the mapping relationship between the original physical address and the fingerprint value of the unit data is established, and the first metadata is stored in the metadata management module for deduplication operation of the unit data.

In one or more embodiments, the deduplication judging module 20 includes a fingerprint value and logical address judging module configured to judge whether the unit data has a corresponding fingerprint value and logical address; Address, to confirm that the unit data has not been deduplicated; in response to the unit data having a corresponding fingerprint value and no corresponding logical address, confirming that the unit data has been deduplicated.

In one or more embodiments, the system further includes a second metadata group storage module configured to, in response to the existence of original metadata, establish a second metadata group including a mapping relationship between the original physical address and the logical address of the unit data , and store the second metadata group in the metadata management module.

In one or more embodiments, the system further includes a logical address invalidation module configured to invalidate the logical address of the unit of data in response to storing the second metadata group into the metadata management module.

In one or more embodiments, the system further includes a garbage collection module configured to notify the garbage collection module to perform garbage collection on the unit data in response to storing both the first metadata and the second metadata group in the metadata management module .

In one or more embodiments, the second metadata group storage module includes a key-value pair module configured to create metadata containing a key-value pair of a logical address pointing to a unit of data from an original physical address, and containing The metadata of the key-value pair whose logical address points to the original physical address.

In one or more embodiments, the system further includes a metadata storage module configured to, in response to the absence of original metadata, establish third metadata including the mapping relationship between the fingerprint value of the unit data and its physical address, and establish The fourth metadata of the mapping relationship between the logical address of the unit data and its physical address is included, and the third metadata and the fourth metadata are stored in the metadata management module.

For specific limitations on the data deduplication device, refer to the above-mentioned limitations on the data deduplication method, which will not be repeated here. Each module in the above-mentioned data deduplication device can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In a third aspect of the embodiments of the present application, one or more non-volatile computer-readable storage media storing computer-readable instructions is also provided. FIG. 3 shows the A schematic diagram of a non-volatile computer-readable storage medium implementing a data deduplication method. As shown in FIG. 3 , the non-volatile computer-readable storage medium 3 stores computer-readable instructions 31 . When the computer-readable instructions 31 are executed by the processor, the method in any one of the above-mentioned embodiments is realized.

It should be understood that all the implementations, features and advantages described above for the data deduplication method according to the present application are equally applicable to the data deduplication system and the storage medium according to the present application if they do not conflict with each other.

The fourth aspect of the embodiments of the present application also provides a computer device, including a memory 402 and one or more processors 401 as shown in FIG. When the read instruction is executed by one or more processors 401, the method in any one of the foregoing embodiments is implemented.

As shown in FIG. 4 , it is a schematic diagram of a hardware structure of an embodiment of a computer device performing a data deduplication method provided by one or more embodiments of the present application. Taking the computer equipment shown in FIG. 4 as an example, the computer equipment includes a processor 401 and a memory 402 , and may further include: an input device 403 and an output device 404 . The processor 401, the memory 402, the input device 403, and the output device 404 may be connected via a bus or in other ways. In FIG. 4, connection via a bus is taken as an example. The input device 403 can receive input numbers or character information, and generate key signal input related to user settings and function control of the data deduplication system. The output device 404 may include a display device such as a display screen.

The memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the computer corresponding to the data deduplication method in the embodiment of this application Readable directives/modules. The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created by using the data deduplication method, and the like. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In one or more embodiments, the memory 402 may optionally include memory that is remotely located relative to the processor 401, and these remote memories may be connected to the local module through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor 401 executes various functional applications and data processing of the server by running non-volatile software programs, instructions and modules stored in the memory 402, that is, implements the data deduplication method in the above method embodiment.

Those skilled in the art can understand that the structure shown in Figure 4 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the equipment to which the solution of the application is applied. The specific equipment may include More or fewer components are shown in the figures, or certain components are combined, or have different component arrangements.

Finally, it should be noted that the computer-readable storage medium (eg, memory) herein may be a volatile memory or a nonvolatile memory, or may include both volatile memory and nonvolatile memory. By way of example and not limitation, nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory memory. Volatile memory can include random access memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in various forms such as Synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). Storage devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

Those of skill would also appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as software or as hardware depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the functions in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope disclosed in the embodiments of the present application.

The above are the exemplary embodiments disclosed in the present application, but it should be noted that various changes and modifications can be made without departing from the scope of the embodiments disclosed in the present application defined by the claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present application may be described or required in an individual form, they may also be understood as plural unless explicitly limited to a singular number.

It should be understood that as used herein, the singular form "a" and "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items. The serial numbers of the embodiments disclosed in the above-mentioned embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is exemplary only, and is not intended to imply that the scope (including claims) disclosed by the embodiments of the present application is limited to these examples; under the idea of the embodiments of the present application , the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present application, which are not provided in details for the sake of brevity. Therefore, within the spirit and principle of the embodiments of the present application, any omissions, modifications, equivalent replacements, improvements, etc., shall be included in the protection scope of the embodiments of the present application.

Claims

A data deduplication method is characterized in that, comprising the following steps:

In response to the host giving up the deduplication operation and writing the data in units of granularity to the hard disk, calculating the fingerprint value of the data, and writing the fingerprint value of the data and its logical address into the hard disk, and judging whether the occupied storage space in the hard disk reaches a preset threshold;

Responding to the occupied storage space reaching a preset threshold, acquiring unit data from the hard disk, and judging whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data;

In response to the fact that the unit data has not been deduplicated, obtain the fingerprint value of the unit data, and query whether there is original metadata including the original physical address that has a mapping relationship with the fingerprint value of the unit data through the metadata management module ;as well as

In response to the presence of the original metadata, establish first metadata including a mapping relationship between the original physical address and the fingerprint value of the unit data, and store the first metadata in the metadata management module for Perform a deduplication operation on the unit of data.
The method according to claim 1, wherein the judging whether the unit data has been deduplicated based on the fingerprint value and/or logical address of the unit data includes:

Judging whether the unit data has a corresponding fingerprint value and logical address;

Responding to the fact that the unit of data has a corresponding fingerprint value and logical address, confirming that the unit of data has not been deduplicated; and

In response to the fact that the unit of data has a corresponding fingerprint value but does not have a corresponding logical address, it is confirmed that the unit of data has undergone a deduplication operation.
The method according to claim 1, further comprising:

In response to the existence of the original metadata, establish a second metadata group including the mapping relationship between the original physical address and the logical address of the unit data, and store the second metadata group in the metadata management module.
The method according to claim 3, further comprising:

In response to storing the second metadata group into the metadata management module, invalidate the logical address of the unit of data.
The method according to claim 3, further comprising:

In response to storing both the first metadata and the second metadata group in the metadata management module, the garbage collection module is notified to perform garbage collection on the unit of data.
The method according to claim 3, wherein the establishment of the second metadata group including the mapping relationship between the original physical address and the logical address of the unit data comprises:

Create metadata including key-value pairs pointing from the original physical address to the logical address of the unit of data, and metadata including key-value pairs pointing to the original physical address from the logical address of the unit of data.
The method according to claim 1, further comprising:

In response to the absence of the original metadata, establishing third metadata including a mapping relationship between the fingerprint value of the unit of data and its physical address, and establishing a mapping relationship between the logical address of the unit of data and its physical address fourth metadata, and store the third metadata and fourth metadata in the metadata management module.
A data deduplication system, comprising:

The storage space judging module is configured to respond to the host giving up the operation of deduplicating data and write the data in the unit of granularity to the hard disk, calculate the fingerprint value of the data, and compare the fingerprint value of the data and its logic writing the address into the hard disk, and judging whether the occupied storage space in the hard disk reaches a preset threshold;

A deduplication judging module, configured to obtain unit data from the hard disk in response to the occupied storage space reaching a preset threshold, and judge whether the unit data has been processed based on the fingerprint value and/or logical address of the unit data Deduplication operation;

The original metadata query module is configured to obtain the fingerprint value of the unit data in response to the fact that the unit data has not been deduplicated, and query whether there is a mapping between the fingerprint value and the unit data through the metadata management module the original metadata of the original physical address of the relationship; and

A data deduplication module configured to, in response to the existence of the original metadata, establish first metadata including a mapping relationship between the original physical address and the fingerprint value of the unit data, and store the first metadata to the metadata management module to perform the deduplication operation of the unit data.
One or more non-transitory computer-readable storage media storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors Carrying out the steps of the method as claimed in any one of claims 1 to 7.
A computer device, characterized by comprising a memory and one or more processors, wherein computer readable instructions are stored in the memory, and when the computer readable instructions are executed by the one or more processors, the The one or more processors execute the steps of the method according to any one of claims 1-7.