JP6163187B2 - Cluster system data processing method and apparatus - Google Patents

Description

  Embodiments described herein relate generally to storage technology, and more particularly, to a data processing method and apparatus in a cluster system.

  Data deduplication (deduplication for short), also called intelligent compression or single-instance storage, automatically searches for duplicate data to meet the need to eliminate redundant data and reduce storage capacity. This is a storage technology that can secure only a unique copy of the data of the other data and replace another duplicate copy by using a pointer that points to a single copy.

  Cluster data deduplication (cluster deduplication for short) represents a technique that organizes multiple deduplication physical nodes to improve deduplication performance and capacity. In the conventional cluster deduplication technology, generally, a physical node that receives a data stream divides the data stream into a plurality of data blocks, groups the obtained data blocks, and for each group, meta data of the data block of the group. A part of the metadata information is sampled from the data information, and a part of the metadata information is transmitted to all the physical nodes of the cluster system for a query. Each physical node of the cluster system stores known data blocks and corresponding metadata information, compares the sampled metadata information with the metadata information stored in each physical node, and determines the most from the query result. A target physical node having a number of duplicate data blocks is obtained, and all data block information of the data group corresponding to the metadata information sampled for the duplicate data query is transmitted to the target physical node.

  Through research, the inventor has shown that in the prior art cluster deduplication technology, the sampled metadata information needs to be sent to all physical nodes for querying, which means that between the physical nodes in the deduplication process When a large number of interaction times are derived and there are many physical nodes in the cluster system, each deduplication of each physical node increases the amount of computation as the number of physical nodes in the cluster system increases, and the system deduplication performance Leading to a decline.

  Embodiments of the present invention provide a data processing method and apparatus in a cluster system to effectively improve the deduplication performance of the cluster system.

In order to solve the problems of the present invention, a first aspect of the present invention is a data processing method in a cluster system,
Dividing the data stream received by the current physical node into m data blocks, obtaining a fingerprint of each data block, and obtaining n first sketch values representing the data stream according to a first algorithm Where m is an integer greater than or equal to 1, n is greater than or equal to 1 and is an integer less than or equal to m, and
Corresponding to each first sketch value of the n first sketch values according to a second algorithm, identifying a first physical node in the cluster system, and assigning each first sketch value to the corresponding first for querying Sending to the physical node;
Receiving at least one response message from a first physical node corresponding to the n first sketch values and obtaining a first storage address from the response message;
The first storage address is obtained from all the received storage addresses, the fingerprint of the data block stored in the storage area indicated by the first storage address, and each data block obtained through the division of the data stream Querying duplicate data blocks, wherein the obtained non-duplicate data block is used as a new data block, and querying
Obtaining at least one second sketch value representing the new data block, identifying a second physical node corresponding to each second sketch value according to the second algorithm by the second sketch value, and each second sketch value; Storing a corresponding relationship between the storage address of the data block represented by each second sketch value in the corresponding second physical node;
A method is provided.

  In connection with the first aspect, in a first possible method of the first aspect, the step of obtaining the first storage address from all received storage addresses comprises the storage corresponding to the first sketch value from the response message. Obtaining an address, collecting statistics on the number of hits of each storage address conveyed in the response message, and selecting s storage addresses having the maximum number of hits as the first storage address, where s is 1 That's it.

  In connection with the first possible method of the first aspect, in the second possible method, the method further comprises: when a predetermined storage condition is met, the new data block and the new data according to a predetermined storage policy. Selecting a storage address to which a new data block fingerprint is written, and writing the new data block and the new data block fingerprint to a storage area indicated by the selected storage address. .

  In connection with the second possible method of the first aspect, in a third possible method, if a predetermined storage condition is met, the new data block is stored in a cache of the current physical node. And the predetermined storage condition is satisfied when the cache data of the current physical node reaches a second predetermined threshold.

A second aspect of the embodiment of the present invention is a data processing apparatus in a cluster system,
A dividing unit that divides a data stream received by a current physical node into m data blocks and obtains a fingerprint of each data block, wherein m is an integer greater than or equal to 1,
A sketch value acquisition unit for acquiring n first sketch values representing the data stream according to a first algorithm, wherein n is an integer not less than 1 and not more than m;
A specifying unit for specifying a first physical node in the cluster system corresponding to each of the first sketch values of the n first sketch values according to a second algorithm;
A transmitter for transmitting each first sketch value to the corresponding first physical node for a query;
A receiving unit that receives at least one response message from a first physical node corresponding to the n first sketch values, and acquires a first storage address from the response message;
A query that queries the duplicate data block by comparing the fingerprint of the data block stored in the storage area indicated by the first storage address with the fingerprint of each data block obtained through the division of the data stream A non-overlapping data block acquired by the query unit is used as a new data block; and
Obtaining at least one second sketch value representing the new data block, identifying a second physical node corresponding to each second sketch value according to the second algorithm by the second sketch value, and each second sketch value; And an update unit that stores a corresponding relationship between the storage address of the data block represented by each second sketch value in the corresponding second physical node;
An apparatus is provided.

In connection with the second aspect, in a first possible realization method, the receiving unit comprises:
A statistics collection sub-unit configured to collect statistics regarding the number of hits of each storage address carried in the response message;
An address acquisition sub-unit configured to select s storage addresses having a maximum hit count as the first storage address according to the statistical result of the statistic collection sub-unit, wherein s is 1 or more, An acquisition sub-part,
Have

In connection with the second aspect and the first possible method of the second aspect, in a second possible implementation method, the apparatus further comprises:
A write address acquisition unit configured to select a storage address to which the new data block and the fingerprint of the new data block are written according to a predetermined storage policy;
A write unit configured to write the new data block and the fingerprint of the new data block to a storage area corresponding to the selected storage address when a predetermined storage condition is satisfied;
It has further.

  In connection with the second possible method of the second aspect, in a third possible implementation method, the writing unit is configured to store the new data block in a cache of the current physical node; When the cache data of the current physical node reaches a second predetermined threshold and the predetermined storage condition is satisfied, the new data block and the fingerprint of the new data block are changed to the selected storage address. Is configured to write to a storage area corresponding to

A third aspect of the embodiment of the present invention is a data processing apparatus in a cluster system, further comprising a processor, a memory, and a bus,
The processor and the memory communicate with each other via the bus;
The memory is configured to store a program;
The processor executes the program in the memory;
The program is
A dividing unit that divides a data stream received by a current physical node into m data blocks and obtains a fingerprint of each data block, wherein m is an integer greater than or equal to 1,
A specifying unit for specifying a first physical node in the cluster system corresponding to each of the first sketch values of the n first sketch values according to a second algorithm;
A transmitter for transmitting each first sketch value to the corresponding first physical node for a query;
A receiving unit that receives at least one response message from the first physical node and obtains a first storage address from the response message;
A query that queries the duplicate data block by comparing the fingerprint of the data block stored in the storage area indicated by the first storage address with the fingerprint of each data block obtained through the division of the data stream A query part in which the acquired non-overlapping data block is used as a new data block;
Obtaining at least one second sketch value representing the new data block, identifying a second physical node corresponding to each second sketch value according to the second algorithm by the second sketch value, and each second sketch value; And an update unit that stores a corresponding relationship between the storage address of the data block represented by each second sketch value in the corresponding second physical node;
A data processing apparatus is provided.

In connection with the third aspect, a first possible implementation method comprises:
The receiver is
A statistics collection sub-unit configured to obtain a storage address corresponding to the first sketch value from the response message and collect statistics about the number of hits of each storage address carried in the response message;
An address acquisition sub-unit configured to select s storage addresses having the maximum number of hits as the first storage address according to a statistical result of the statistic collection sub-unit, wherein s is 1 or more; An acquisition sub-part,
Have

In connection with the second aspect and the first possible method of the second aspect, in a second possible implementation method, the apparatus further comprises:
A write address acquisition unit configured to select a storage address to which the new data block and the fingerprint of the new data block are written according to a predetermined storage policy;
A write unit configured to write the new data block and the fingerprint of the new data block to a storage area corresponding to the selected storage address when a predetermined storage condition is satisfied;
It has further.

  In connection with the second possible method of the second aspect, in a third possible implementation method, the writing unit is configured to store the new data block in a cache of the current physical node; When the cache data of the current physical node reaches a second predetermined threshold and the predetermined storage condition is satisfied, the new data block and the fingerprint of the new data block are changed to the selected storage address. Is configured to write to a storage area corresponding to

A fourth aspect of an embodiment of the present invention is a computer program for data processing, the computer program having a computer-readable storage medium that stores program code, and instructions included in the program code include:
Dividing the data stream received by the current physical node into m data blocks, obtaining a fingerprint of each data block, and obtaining n first sketch values representing the data stream according to a first algorithm Where m is an integer greater than or equal to 1, n is greater than or equal to 1 and is an integer less than or equal to m, and
Corresponding to each first sketch value of the n first sketch values according to a second algorithm, identifying a first physical node in the cluster system, and assigning each first sketch value to the corresponding first for querying Sending to the physical node;
Receiving at least one response message from a first physical node corresponding to the n first sketch values and obtaining a first storage address from the response message;
The first storage address is obtained from all the received storage addresses, the fingerprint of the data block stored in the storage area indicated by the first storage address, and each data block obtained through the division of the data stream Querying duplicate data blocks, wherein the obtained non-duplicate data block is used as a new data block, and querying
Obtaining at least one second sketch value representing the new data block, identifying a second physical node corresponding to each second sketch value according to the second algorithm by the second sketch value, and each second sketch value; Storing a corresponding relationship between the storage address of the data block represented by each second sketch value in the corresponding second physical node;
Used to execute

  In an embodiment of the present invention, when a duplicate data query is performed on a received data stream, a first physical node corresponding to each first sketch value and in the cluster system represents a first representing the data stream. A first sketch value, identified according to the sketch value, is then sent to the identified physical node for the duplicate data query, and the number of identified physical nodes for performing the duplicate data query is the cluster The increase in the number of nodes in the system does not increase, and the calculation amount of each node does not increase due to the increase in the number of nodes in the cluster system, and the deduplication performance of the cluster system is effectively improved.

In order to more clearly describe the technical means in the embodiments of the present invention or in the prior art, the following briefly introduces the accompanying drawings required for describing the embodiments or the prior art. Apparently, the accompanying drawings in the following description merely show some embodiments of the present invention, and those skilled in the art may still derive other drawings from the accompanying drawings without creative efforts.
FIG. 1 is a flowchart of an embodiment of a data processing method in a cluster system according to the present invention. FIG. 2 is a schematic diagram of block division and sampling performed by a physical node after receiving a data stream according to an embodiment of the present invention. FIG. 3 is a schematic diagram of an internal structure of a physical node according to an embodiment of the present invention. FIG. 4 is a flowchart of an embodiment of another data processing method in the cluster system according to the present invention. FIG. 5 is a flowchart of an embodiment of another data processing method in the cluster system according to the present invention. FIG. 6 is a schematic diagram of the internal structure of another physical node according to an embodiment of the present invention. FIG. 7 is a flowchart of an embodiment of another data processing method in the cluster system according to the present invention. FIG. 8 is a schematic diagram of data migration according to an embodiment of the present invention. FIG. 9 is a configuration diagram of a data processing apparatus in the cluster system according to the embodiment of the present invention. FIG. 10 is a configuration diagram of another data processing apparatus in the cluster system according to the embodiment of the present invention.

  DESCRIPTION OF EMBODIMENTS The following clearly and completely describes the technical means in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are merely a part rather than all of the embodiments of the present invention. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

  The embodiments of the present invention can be applied to a cluster system for storage in which the cluster system includes a plurality of physical nodes. The physical node having the deduplication engine may be used as an execution subject of the embodiment of the present invention, and executes the method of the embodiment of the present invention after obtaining the deduplication task.

  FIG. 1 is a flowchart of an embodiment of a data processing method in a cluster system according to the present invention. As shown in FIG. 1, the method of this embodiment may include the following.

  Step 10: Divide the data stream received by the current physical node into m data blocks and obtain the fingerprint of each data block. However, m is an integer greater than or equal to 1.

  In a cluster system, a physical node may be used as a node for executing a cluster deduplication task as long as an apparatus that executes the data processing method according to the embodiment of this invention is installed in the physical node.

  After receiving the data stream, the current physical node divides the data stream data into m data blocks and obtains a fingerprint of each data block.

  Step 11: Obtain n first sketch values representing a data stream according to a first algorithm. However, n is an integer of 1 or more and m or less.

  N first sketch values representing a data stream are obtained via a predetermined algorithm, and for convenience of explanation, an algorithm for obtaining a first sketch value representing a data stream is called a first algorithm and received. The sketch value representing the data stream is called the first sketch value.

  The first algorithm is not limited to the embodiment of the present invention as long as it can obtain the first sketch value representing the data stream. For example, a data block obtained through partitioning is directly sampled, sketch values of n sampled data blocks are calculated, and the calculated sketch value is used as a sketch value representing the received data stream.

  Optionally, in order to make the sampled data more uniform, in an embodiment of the present invention, obtaining the n sketch values representing the data stream according to the first algorithm may include the following steps: Good.

  The fingerprints of the data blocks obtained by the received data blocks are grouped into n first fingerprint groups, each first finger group including at least two data block fingerprints.

  For each first fingerprint group, in order to obtain n first sketch values, a first sketch value representing each fingerprint group is obtained.

  If n is equal to 1, it means that the fingerprints corresponding to the data blocks obtained from the received data stream are grouped into one first fingerprint group, and one fingerprint is the data obtained through the division. Sampled directly from the fingerprint corresponding to the block as the first sketch value of the first fingerprint group.

  In an embodiment of the present invention, multiple consecutive data blocks are grouped into one first fingerprint group, and possibly there are multiple methods for dividing the data block to obtain the first fingerprint group. However, the present invention is not limited to this example.

  In an embodiment of the present invention, if a first sketch value is obtained by a method of dividing a data stream into data blocks and grouping fingerprints corresponding to the data blocks, the first sketch value corresponds to the first fingerprint. There may be multiple ways to obtain the first sketch value from each fingerprint group as long as the group can be represented. For example, the largest fingerprint of the data block fingerprints may be selected as the first sketch value, or the fingerprint information of one or more data blocks may be randomly selected as the first sketch value from the first fingerprint group. This is selected and is not limited in the embodiments of the present invention.

  In this embodiment, the fingerprint information of all data blocks of the first fingerprint group is the minimum fingerprint, ie, the minimum fingerprint value is also the first sketch value representing the fingerprint group to which the minimum fingerprint belongs. May be selected.

  Step 12: Identify a first physical node in the cluster system corresponding to each first sketch value of the n first sketch values according to the second algorithm.

  In an embodiment of the present invention, for convenience of explanation, a physical node corresponding to the specified first sketch value is called a first physical node, an algorithm for specifying the first physical node is called a second algorithm, If one first sketch value corresponds to one first physical node, n first sketch values may correspond to n first physical nodes, and embodiments of the present invention are probably limited to this. Not.

  There may be a plurality of second algorithms. For example, route extraction and rounding is performed on the first sketch value until the value is less than m, and the final result is rounded to obtain the corresponding first physical node.

  In the present embodiment, the second algorithm employed further performs a modulo operation with the first sketch value on the number of all physical nodes of the cluster system, corresponding to the first sketch value, and in the cluster system. A certain first physical node is acquired.

  Step 13: Send each first sketch value to the corresponding first physical node for query, receive at least one response message from the first physical node corresponding to the n first sketch values, 1 storage address is acquired.

  In an embodiment of the present invention, in a cluster system, each physical node or most physical nodes stores an index table between the sketch value and the storage address of the stored data block represented by the sketch value. The correspondence relationship is stored in the index table, the index table is stored in the physical node of the cluster system according to a predetermined policy, and the data block and the fingerprint information corresponding to the data block are indicated by different storage addresses. Stored in the area.

  The index table for each physical node may be a complete index table, which is the sum of all the sketch values of the stored data and the storage address of the stored data block represented by that sketch value in the cluster system. It should be noted that a part of the complete index table may also include a corresponding policy between the given policy to reduce the amount of computation for querying duplicate blocks in the physical node. May be stored in each physical node. The latter method may be employed in this embodiment. How each index entry of the index table is stored in the physical node is described in detail below.

  In the embodiment of the present invention, the index table is stored in the physical node according to a predetermined policy, and there is no direct relationship between the data of the index table of the specific physical node and the data of the physical node, only the index table. Are stored in the physical node according to a predetermined allocation policy.

  In a specific implementation, since a storage area corresponding to one storage address may have data of a plurality of groups, when a sketch value is selected from each group, one storage address has a plurality of different sketches. Cases may occur that correspond to values, and therefore the same storage address in the index table may correspond to multiple different sketch values. However, one identical sketch value corresponds to one storage address. When a plurality of first sketch values are queried in the index table, a plurality of corresponding storage addresses may be obtained, and when the first storage address is selected from among the storage addresses corresponding to the first sketch values, the first A rule for selecting one storage address may be set in advance, or may be set by a user according to an actual situation, and is not limited in this embodiment.

  For example, the storage address corresponding to the first sketch value may be obtained from the response message, statistics regarding the number of hits of each storage address conveyed in the response message are collected, and s storages having the maximum number of hits An address is selected as the first storage address. However, s is 1 or more.

  In a specific implementation, upon receiving a query request carrying the first sketch value, the first physical node matches the same sketch value as the first sketch value according to a locally stored index table and sends a response message. The storage address corresponding to the matched sketch value is fed back to the physical node that transmitted the query request. Each first sketch value corresponds to one first physical node, and thus a plurality of first physical nodes receive a query request and execute the query locally according to the query request. If the storage address is detected via a query, the storage address is returned to the node that sent the query request, and the node that sent the query request may receive multiple storage addresses returned by different first physical nodes. Well, if the storage address was not found via a query, for example, a null value such as 0 is fed back or no feedback is returned, specifically indicating that the storage address was not found via a query The method may be set by the user.

  If the storage address fed back by the first physical node is not received after each first sketch value has been sent to the corresponding first physical node for a query, the data block obtained via the splitting of the data stream is It is used as a non-overlapping data block, after which step 15 is executed.

  Step 14: comparing the fingerprint of the data block stored in the storage area indicated by the first storage address with the fingerprint of each data block obtained through the division of the received data block, Query. However, the acquired non-overlapping data block is used as a new data block.

  In a specific implementation, the current physical node stores the data block and fingerprint information in a cache after processing the received data stream, and compares the fingerprint of the data block using multiple methods. Also good. For example, in Method 1, the storage area data block corresponding to the first storage address is loaded into the cache of the current physical node, and the fingerprint of each data block corresponding to the received data stream is Executed. In method 2, the query instruction is transmitted to the physical node indicated by the first storage address. Here, the fingerprint of the m data blocks obtained through the division of the data stream is conveyed to the query instruction, and the query result returned by the physical node indicated by the first storage address is received. In Method 1, fingerprint information stored in the current node's cache may be utilized in the duplicate data query, which may further improve the deduplication rate.

  When an index entry in the index table is initialized, the user can also determine between the known sketch value and the storage address of the data block represented by the known sketch value according to the data stored in the system in the index table. These correspondence relationships may be stored in advance, and a specific method is not limited in the embodiments of the present invention. The index table may also be null during initialization, during a duplicate data query, when a new data block acquired is stored in the storage system, a new sketch represented by the acquired sketch value and the sketch value. The storage address of the data block is inserted into the index table, whereby the index table is periodically updated. Accordingly, embodiments of the present invention further include:

  Step 15: Obtain at least one second sketch value representing a new data block, identify a second physical node corresponding to each second sketch value according to a second algorithm by the second sketch value, and each second sketch value And the corresponding relationship between the storage address of the data block represented by each second sketch value is stored in the corresponding second physical node.

  In an embodiment of the present invention, a non-overlapping data block detected through a query is used as a new data block, and the new data block detected through the query reaches a certain size and needs to be stored The storage location of new data blocks in the system needs to be indicated in the index table, and the index table needs to be updated regularly. When a storage address is obtained for a new data block, a sketch value representing the new data block is obtained, referred to as a second sketch value for convenience of explanation, a second physical node is identified according to a second algorithm, Correspondence between each second sketch value and the storage address of the data block represented by each second sketch value is stored in the corresponding second physical node, so that the periodic entry of the index entry of the cluster system Update can be realized.

  The method for obtaining the second sketch value may be the same as the algorithm for obtaining the first sketch value. For example, a first algorithm may be employed, if the first set of sketch values is guaranteed to be a subset of the second set of sketch values, or the second set of sketch values is a set of first sketch values. Other algorithms may be employed.

  The means of this embodiment will be described in detail below with reference to specific examples of specific implementation methods. FIG. 2 is a schematic diagram of block partitioning and sampling performed by the current physical node after reception of the data stream. In this embodiment, a plurality of data blocks may be regarded as one super data block (super chunk). In a specific implementation, fingerprints corresponding to data blocks are grouped for more accurate grouping. The data block corresponding to the fingerprint of the group is a super data block, and the first sketch value may also be referred to as a super data block sketch value (super chunk ID, SID). Perhaps data blocks obtained via splitting of the data stream may also be grouped directly. Referring to FIG. 2, after receiving the data stream, the current physical node divides the received data into data blocks, and a plurality of consecutive data blocks are grouped into one group, i.e. Grouped into chunks. Grouping methods include methods such as variable-length block division and fixed-length block division, and the present embodiment is not limited to this. A data block is acquired from each super chunk, and a fingerprint corresponding to the acquired data block is used as an SID representing each super chunk.

  By extracting physical nodes of the cluster system as an example, reference may be made to FIG. 3 for a schematic diagram illustrating an example of the internal structure of the physical nodes. A cache and a storage area may be included in each physical node. In a specific implementation, an index table is stored in the physical node cache to improve clear performance, and a second sketch value and the second sketch are stored. Correspondence between the storage address of the stored data block represented by the value is stored in the index table. As described above, the index table of each physical node may be a complete index table, and the stored data block fingers represented by all the second sketch values and the second sketch values in the cluster system. Includes the correspondence between print storage addresses. A portion of the complete index table may also be stored at each physical node according to a predetermined policy. In a specific implementation, the latter method may be employed to reduce memory usage, and a portion of the complete index table is stored in each physical node according to a predetermined policy. For example, by extracting the sketch value A as an example, a physical node is identified by performing a modulo operation with A on the sum of all physical nodes of the cluster system. The correspondence between the storage address of the stored data block represented by A and A is stored in the identified physical node, and the first physical node corresponding to the first sketch value is obtained during the duplicate data query Therefore, a modulo operation using the acquired first sketch value is performed on the total of physical nodes of the cluster system.

  Each physical node has a storage device, which allows each physical node to have a function of storing data for a long time, which may be a magnetic disk, and other storage devices such as SSDs It may be. The storage device of each physical node is called a single instance repository (single instance repository SIR). In FIG. 3, a magnetic disk is taken as an example. The storage device of the physical node may have a large number of storage areas, and each storage area may be regarded as an example of a container for storing data, and each storage container is one unique in the cluster system. A serial number, which may be referred to as a storage container serial number (container IDCID). The container serial number indicates the position of the storage container in the cluster system such as which storage area of which physical node of the cluster system. Therefore, in a specific implementation, the above-described storage address of the stored data block is realized as a CID indicating in which storage address of which physical node the data block is stored. The correspondence relationship between the storage address of the stored data block represented by the sketch value in the index table may be shown as the correspondence relationship between the SID and the CID, in addition to the storage data block. The fingerprint information corresponding to the data block may be further stored in each storage area. A buffer storage area (container buffer) may further be included in the cache of each physical node, and is configured to temporarily store new data blocks obtained via identification.

  In the prior art, the sampled fingerprint information is sent to all nodes for a query. Thus, when the number of nodes in the cluster system increases, the amount of calculation becomes larger. Experiments show that when the number of nodes in a cluster system exceeds 16, the deduplication performance of the system is significantly reduced. In an embodiment of the present invention, unlike the prior art where sampled sketch values are sent to all nodes for a query, each first sketch is executed when a duplicate data query is performed on the received data stream. A first physical node corresponding to the value and within the cluster system is identified according to a first sketch value representing the data stream, after which the first sketch value representing the data stream is identified for the duplicate data query. The duplicate data query procedure does not change with an increase in the number of nodes in the cluster system, and the calculation amount of each node does not increase with an increase in the number of nodes in the cluster system.

  Referring to FIG. 4, the present invention further provides an embodiment of another cluster data processing method. The difference from the data processing method in the cluster system corresponding to FIG. Thereafter, it is determined whether or not there is a new data block in the received data stream, and it is required to store the new data block, and the other steps are the same as those in the embodiment corresponding to FIG. Thus, based on the embodiment corresponding to FIG. 1, the method may further include:

  Step 46: Select a storage address to which a new data block and a fingerprint of the new data block are written according to a predetermined storage policy.

  The storage policy for a new data block may be set in advance by the user, or there may be a plurality of policies. For example, in policy 1, the load information of each physical node of the cluster system is first acquired, the physical node to which data is migrated is selected according to the load information, and the new data block and the fingerprint of the new data block are written Storage addresses are obtained from the selected physical node to which the data is migrated, and thus load balancing is performed on all physical nodes. In policy 2, the load information does not have to be determined, and the storage address to which the new data block and the fingerprint of the new data block are written is obtained directly from the current physical node, thus Interaction is reduced. Therefore, the specific storage policy may be set by the user according to the actual situation, and is not limited in the embodiment of the present invention.

  If policy 1 is employed, there may be multiple ways of writing data according to the load information. For example, a threshold value may be set in advance and referred to as a first predetermined threshold value. When the average load value of all the physical nodes of the cluster system exceeds the first predetermined threshold, a node to which data is written is selected from physical nodes having a load value smaller than the average load value, and a new data block and the new load block are selected. When the storage address to which the fingerprint of the data block is written is obtained from the node to which the data is written, and the average load value of all the physical nodes of the cluster system is equal to or less than the first predetermined threshold, the new data block and the new data block The storage address to which the fingerprint of the correct data block is written is obtained from the current physical node.

  Other methods for performing data migration according to load values may include:

  A1: When the load information of the physical node of the cluster system is acquired and the load difference between any two physical nodes of the cluster system is larger than the first threshold, the node to which the data is migrated is the physical node with the lower load The storage address to which the new data block and the fingerprint of the new data block are written is acquired from the node to which the data is migrated.

  A2: When the load difference between any two physical nodes of the cluster system is less than or equal to the first threshold value, the storage address where the new data block and the fingerprint of the new data block are written from the current physical node get.

  The load information may be obtained from the current physical node, and the load information may also be obtained directly from the third party by the current physical node after the third party obtains the load information.

  Step 47: When a predetermined storage condition is satisfied, a new data block and a fingerprint of the new data block are written in the storage area corresponding to the selected storage address.

  Optionally, the condition for a given storage is met when a new data block is stored in the current physical node's cache and when the current physical node's cache data reaches a second predetermined threshold, It may be included that the storage condition is satisfied.

  As illustrated in the specific example of FIG. 3, a container buffer may be included in each physical node, and is configured to temporarily store a new data block obtained through identification and store in the container buffer. When the size of the data to be exceeded exceeds the second threshold, it is considered that a predetermined storage condition has been satisfied, and the user may set the second threshold according to the actual situation. This is not limited in this embodiment.

  In the embodiment of the present invention, after the storage address of a new data block is acquired, the second sketch value and the storage address in the corresponding second physical node of the data block represented by the second sketch value are obtained. The specific opportunity for storing the correspondence relationship is not limited in the embodiment of the present invention.

  In the data processing method in the cluster system provided by the embodiment corresponding to FIG. 4, when a new data block is stored, load balancing is realized and system performance is improved.

  Based on the data processing method corresponding to FIG. 1, and referring to FIG. 5, the present invention provides another embodiment for the case where the virtual node is included in the cluster system. At least one virtual node is logically obtained through partitioning of the physical nodes of the cluster system, and the correspondence (VPT) between all virtual nodes of the cluster system and all physical nodes is stored in each physical node. In an embodiment of the present invention, it is assumed that at least one virtual node is logically obtained through partitioning of each physical node. The duplicate data query method includes:

  Step 50: Process the data stream received by the current physical node to obtain m data blocks and obtain the fingerprint of each data block.

  Step 51: Obtain n first sketch values representing a data stream according to a first algorithm. Here, n is an integer of 1 or more and m or less.

  The method for processing the data stream and the method for obtaining the first sketch values in steps 50 and 51 are the same as in the embodiment corresponding to FIG. 1 and will not be described again here.

  Step 52: Corresponding to each sketch value of the n first sketch values according to the second algorithm, identifying the first virtual node in the cluster system, and correspondence between the virtual node of the current physical node and the physical node The first physical node corresponding to the first virtual node is obtained by querying the relationship. Here, each first sketch value may correspond to one first virtual node, and one first virtual node may simultaneously correspond to a plurality of first sketch values.

  For convenience of explanation, the corresponding virtual node identified according to the first sketch value is referred to as the first virtual node, and at least one virtual node may be obtained through the division of each physical node of the cluster storage system, specifically In a typical implementation, two or more virtual nodes may be obtained through partitioning of each physical node, each virtual node having a unique serial number, and between virtual nodes and physical nodes in a cluster system A correspondence relationship may be stored in each physical node, which indicates which physical node the virtual node is on.

  Step 53: Send each first sketch value to the first virtual node of the corresponding first physical node for the query, receive a response message from the first physical node, and store corresponding to the first sketch value from the response message The address is acquired, and the first storage address is selected from the storage addresses corresponding to the first sketch value.

  In an embodiment of the present invention, each physical node or most physical nodes of the cluster system stores an index table, and a sketch value acquired in advance and a storage address of a stored data block represented by the sketch value Are stored in the index table. The index table is stored in the physical node of the cluster system according to a predetermined policy, and after the virtual node is acquired through physical node division, the index table is stored in the virtual node acquired through physical node division. Also good. The method for storing each index entry in the index table in the virtual node is described in detail below.

  When there is no virtual node in the cluster system of the data block, the storage address of the index table specifically indicates any storage area of any physical node, and when there is a virtual node in the cluster system, the storage address is Specifically, it indicates any storage area of any virtual node.

  Similarly, the index table for each virtual node may be a complete index table, between all sketch values and the storage address of the fingerprint of the stored data block represented by the sketch values in the cluster system. Part of the complete index table may also be stored in each virtual node according to a predetermined policy to reduce memory usage.

  A first virtual node in the cluster system corresponding to each of the n first sketch values is identified according to the second algorithm. Similarly, there are a plurality of second algorithms as long as it can be useful for the first sketch value to correspond to one unique virtual node of the cluster system. For example, by extracting the first sketch value A as an example, when a duplicate data query is performed on a data block, all the cluster systems are retrieved to obtain the first virtual node corresponding to the first sketch value. A modulo operation is performed on the first sketch value obtained for the number of virtual nodes, and then the first physical node corresponding to the first virtual node is identified according to the correspondence between the virtual node and the physical node. The When the first physical node is detected, the first virtual node of the first physical node is detected.

  A large number of storage areas are included in each virtual node, and in a specific implementation, each storage area may be considered illustratively as a container for storing data, each container being a unique in the cluster system. Serial number CID, and this number indicates in which storage area of which virtual node of the cluster system the storage area is arranged. In addition to the data blocks, fingerprint information corresponding to the data blocks may also be stored in each storage area.

  A buffer storage area (container buffer) may also be included in each physical node's cache, and is configured to temporarily store new data blocks obtained via identification.

  Similar to the embodiment corresponding to FIG. 1, in a specific implementation, one identical storage address in the index table may correspond to multiple different sketch values, but one sketch value is one storage address. Corresponding to When a plurality of first sketch values are queried in the index table, a plurality of corresponding storage addresses may be obtained, and when the first storage address is selected from the storage addresses corresponding to the first sketch values, the first A rule for selecting a storage address may be set in advance, or may be set by a user according to an actual situation. Reference is made to the description of selecting the first storage address in the method corresponding to FIG. 1, but this is not limited in this embodiment.

  In an embodiment of the present invention, after receiving the query request, the physical node executes the query in the local virtual node, and if the storage address is detected via the query, the physical node sends the query request to the node that transmitted the query request. Provide feedback on query results. In a specific implementation, after receiving a query request, the physical node executes the query in the local virtual node and places the query result in the response message, thereby feeding back the query result to the physical node that sent the query request. To do. If the storage address is detected via a query, the storage address detected via the query is placed in the response message, and if the storage address is not detected via the query, a null value is fed back by the response message, or In a typical implementation, no feedback is returned if the storage address is not found via a query. The user may set a specific implementation method according to the actual situation.

  If the current physical node does not receive the storage address returned by the first physical node corresponding to the n first sketch values, the data block obtained through the division of the data stream is used as a non-overlapping data block. The

  Step 54: Compare the fingerprint of the data block stored in the storage area indicated by the first storage address with the fingerprint of each data block obtained through the division of the received data stream, and duplicate data blocks Query. Here, the acquired non-overlapping data block is used as a new data block.

  In a specific implementation, the current physical node employs multiple methods for storing data blocks and fingerprint information in a cache after processing the received data stream and comparing the fingerprints of the data blocks. The description of the method corresponding to FIG. 1 may be referred to.

  Step 55: Obtain at least one second sketch value representing a new data block, identify a second virtual node corresponding to each second sketch value according to a second algorithm by the second sketch value, and a virtual node and a physical node The second physical node in which the second virtual node is arranged is identified according to the correspondence relationship between the second sketch value and the storage relationship of the storage address of the data block represented by the second sketch value. Store in the second virtual node of the corresponding second physical node.

  Assuming that two virtual nodes are obtained by way of example by dividing physical nodes of the cluster system, reference is made to FIG. 6 of a schematic diagram illustrating an example of the internal structure of the physical nodes. In FIG. 6, two virtual nodes (VN) are acquired through division of one physical node (PN), and an index table and a storage device capable of storing long-term data are included in one virtual node. A correspondence table (VPT) between the virtual node and the physical node is simultaneously stored in each physical node.

  In the prior art, sampled fingerprint information is sent to all nodes for a query. The amount of computation for executing a query at a node is z, and when there are m physical nodes, it is assumed that the amount of computation is z multiplied by m. In the deduplication method in the cluster system having virtual nodes provided by the present embodiment, when the duplicate data query is executed on the received data stream, the first sketch value corresponding to each first sketch value is stored in the cluster system. A physical node is identified according to a first sketch value that represents the data stream, and then the first sketch value that represents the data stream is sent to the identified virtual node for duplicate data queries. The number of identified virtual nodes does not increase with the increase in the number of physical nodes in the cluster system, and the duplicate data query procedure effectively improves the deduplication performance of the cluster system. It does not change with the increase.

  Referring to FIG. 7, the present invention further provides an embodiment of the data processing method in the cluster system, and the difference from the data processing method in the cluster system corresponding to FIG. After that, it is determined whether there is a new data block in the received data stream, and it is required to store the new data block. The other steps are the same as those in the embodiment corresponding to FIG. Thus, based on the embodiment corresponding to FIG. 5, the method may further include:

  Step 76: Select a storage address where a new data block and a fingerprint of the new data block are written according to a predetermined storage policy.

  The storage policy for a new data block may be preset by the user, or there may be a plurality of storage policies. Reference may be made to the description of the method for selecting a storage address where a new data block and a fingerprint corresponding to the new data block are written in the embodiment corresponding to FIG. Once determined, the load of the virtual node within the physical node is considered and the storage address where the new data block and the fingerprint of the new data block are written is loaded at the selected physical node where the data is migrated Is obtained from a lower virtual node.

  Step 77: When a predetermined storage condition is satisfied, a new data block and a fingerprint of the new data block are written into the storage area corresponding to the selected storage address.

  As illustrated in the specific example of FIG. 6, a container buffer may be included in each physical node, and is configured to temporarily store a new data block obtained through identification and stored in the container buffer. If the size of the stored data exceeds the second threshold value, it may be considered that the predetermined storage condition has been satisfied, and the user may set the second threshold value according to the actual situation. It is not limited in.

  Specifically, after the storage address of the new data block is acquired, the correspondence relationship between the second sketch value and the storage address of the data block represented by the second sketch value is stored in the corresponding second physical node. This opportunity is not limited in the embodiments of the present invention.

  In the data processing method in the cluster system provided by the embodiment corresponding to FIG. 7, the index table of the virtual node is periodically updated to further improve the probability of detecting duplicate data.

  Referring to FIG. 8, a means for data migration in a physical node of the cluster system is provided based on the fact that the virtual node is included in the cluster system. For example, FIG. 8 illustrates a schematic diagram of data migration in the case where one physical node is expanded to two physical nodes.

  When the data migration condition is satisfied, one of the virtual nodes of the physical node PN1 is moved to the other physical node PN2 as a whole. Data to PN2, update the correspondence between the migrated virtual node and the physical node in the cluster system, and update the correspondence between the migrated node and the physical node to the other in the cluster system. Including notifying the physical node. The updated correspondences between all virtual nodes and all physical nodes in the cluster system are stored in the newly added PN2.

  The data migration condition may be that the data needs to be migrated to the newly added physical node during the capacity expansion of the physical node of the cluster system, and because of load balancing, The node data may be migrated to a physical node with a low load. Data migration conditions are not limited in the embodiments of the present invention.

  When the virtual node is not acquired through the division in the cluster system and the data of the physical node is migrated, the data block and the storage address of the fingerprint of the data block are changed, so the logical address and CID of the data block It is necessary to change the correspondence between the CIDs, and the related CID data is stored in a large number of physical nodes. However, since the data in the virtual node is moved together, the CID of the index table is stored in the virtual node by using the cluster system provided by the embodiment of the present invention during the data migration. As long as the virtual node identifier is not changed, the CID need not be changed, only the correspondence between the associated virtual node and the physical node is changed, which is the process during node expansion. Simplify the procedure and greatly reduce the amount of calculation.

  The embodiments of the present invention are applied in a cluster system for storage, and a plurality of storage nodes are included in the cluster system. The cluster system data processing apparatus provided by the embodiment of the present invention is configured to execute the above-described data processing method of the cluster system, and is set as a physical node of the cluster system or set as a manager of the cluster system. Or may be configured as another node of the cluster system. The specific position where the data processing apparatus is set is not limited in the embodiment of the present invention.

  Referring to FIG. 9, the embodiment of the present invention provides a data processing device 900 in a cluster system, which includes:

  The dividing unit 90 is configured to divide the data stream received by the current physical node into m data blocks and obtain a fingerprint of each data block. Here, m is an integer of 1 or more.

  Sketch value acquisition unit 91 is configured to acquire n first sketch values representing a data stream according to a first algorithm. Here, n is an integer of 1 or more and m or less.

  Reference may be made to the description in the above-described method of the first algorithm for obtaining the first sketch value, which is not repeated here.

  The specifying unit 92 is configured to specify a first physical node in the cluster system corresponding to each first sketch value of the n first sketch values according to the second algorithm.

  There may be many methods by which the specifying unit 92 specifies the first physical node. For example, route extraction and rounding is performed on the first sketch value until the value is less than m, and the final result is rounded to obtain the corresponding first physical node, with each first sketch In order to obtain a first physical node in the cluster system corresponding to the value, a modulo operation with each first sketch value may be performed on the number of all physical nodes in the cluster system. In the embodiment, the specifying unit 92 specifically acquires the first physical node corresponding to each first sketch value and in the cluster system. You may be comprised so that the modulo operation by one sketch value may be performed.

  The transmitter 93 is configured to transmit each first sketch value to the corresponding first physical node for a query.

  After receiving the query request transmitted by the transmitter 93, the other nodes in the cluster system query whether there is a storage address corresponding to the first sketch value according to the index table stored locally. In an embodiment of the present invention, each physical node or most physical nodes of the cluster system stores an index table, and correspondence between the sketch value and the storage address of the stored data block represented by the sketch value. The relationship is stored in the index table, the index table is stored in the physical node of the cluster system according to a predetermined policy, and the data block and the fingerprint corresponding to the data block are stored in the storage area indicated by different storage addresses Is done.

  The receiving unit 94 is configured to receive at least one response message from the first physical node corresponding to the n first sketch values and acquire the first storage address from the response message.

  The reception unit 94 obtains a storage address corresponding to the first sketch value from the response message, and collects a statistic regarding the number of hits of each storage address carried in the response message; And an address acquisition sub-unit 942 configured to select s storage addresses having the maximum number of hits as the first storage address according to the statistical result of the statistics collection sub-unit 941, where s is 1 or more.

  The query unit 95 compares the fingerprint of the data block stored in the storage area indicated by the first storage address with the fingerprint of each data block obtained through the division of the received data stream, and duplicate data A non-overlapping data block configured to query the block and acquired by the query unit 95 is used as a new data block.

  The update unit 96 acquires at least one second sketch value representing a new data block, specifies a second physical node corresponding to each second sketch value according to the second algorithm, using the second sketch value, and sets each second sketch value. A correspondence relationship between the sketch value and the storage address of the data block represented by each second sketch value is configured to be stored in the corresponding second physical node.

  The algorithm for acquiring the second sketch value representing the new data block by the updating unit 96 is not limited here, and may be the same as the first algorithm for acquiring the first sketch value.

  The correspondence between each second sketch value and the storage address of the data block represented by each second sketch value is used as one index entry in the index table. In the embodiment of the present invention, the update unit 96 stores the index table in the physical node according to the set policy, and there is no direct relationship between the data of the index table of a certain physical node and the data of the physical node. Instead, the index table is only stored in the physical node according to the set allocation policy. Reference may be made to the description of the method described above for the description of the index table, and details are not repeated here.

  With reference to the above description of the method, in a specific implementation, there may be a plurality of methods by which the sketch value acquisition unit 91 acquires n first sketch values representing a data stream according to the first algorithm. For example, the sketch value acquisition unit 91 may include a fingerprint grouping sub unit 911 and an acquisition sub unit 912.

  The fingerprint grouping sub-unit 911 is configured to group the fingerprints of the m data blocks of the data block into n first fingerprint groups, each first fingerprint group including at least two data blocks. With a fingerprint of

  The acquisition sub-unit 912 is configured to acquire a first sketch value representing each fingerprint group in order to acquire n first sketch values.

  Specifically, the acquisition sub-unit 912 may be configured to select the smallest fingerprint from each fingerprint group as the first sketch value representing the fingerprint group to which the smallest fingerprint belongs.

  The receiving unit 94 of the data processing apparatus of the cluster system further transmits the first sketch value to the corresponding first physical node for a query, and if the storage address returned by the first physical node is not received, The data block obtained through the division may be used as a non-overlapping data block, and then the update unit 96 may be triggered.

  An apparatus provided by an embodiment of the present invention may be configured in a controller of a storage node of a cluster system and is configured to execute the data processing method described above. For a detailed description of the function of all units, reference may be made to the description in the method embodiment, and details will not be repeated here.

  When a duplicate data query is executed on the received data stream through the data processing device in the cluster system provided by the embodiment of the present invention, the first physical corresponding to each first sketch value and in the cluster system. A node is identified according to the first sketch value representing the data stream, and then the first sketch value representing the data stream is sent to the identified physical node for the duplicate data query. The duplicate data query procedure does not change with an increase in the number of nodes in the cluster system, and the calculation amount of each node does not increase with an increase in the number of nodes in the cluster system.

  When non-duplicate data is detected through a query, the query unit 95 may regard the non-duplicate data as a new data block and store the new data block in a single instance repository. Accordingly, embodiments of the present invention may further include:

  The write address acquisition unit 97 is configured to select a storage address to which a new data block and a fingerprint of the new data block are written according to a predetermined storage policy.

  The writing unit 98 is configured to write a new data block and a fingerprint of the new data block in a storage area corresponding to the selected storage address when a predetermined storage condition is satisfied.

There may be a plurality of storage policies for new data blocks in the write address acquisition unit 97. For example, the write address acquisition unit 97
A load information acquisition sub-unit 971 configured to acquire load information of each physical node of the cluster system;
According to the load information, the physical node to which data is migrated is selected, and the storage address where the new data block and the fingerprint of the new data block are written is acquired from the selected physical node to which data is migrated An address selection sub-unit 972 configured;
Have

  There may be a plurality of methods by which the address selection sub-unit 972 selects a storage address to which new data is written according to the load information. For example, when it is specified according to the load information acquired by the load information acquisition sub-unit 971 that the average load value of all the physical nodes of the cluster system has exceeded the first predetermined threshold, the node to which the data is written A storage address selected from a physical node having a load value smaller than the load value and into which a new data block and a fingerprint of the new data block are written is obtained from the node to which the data is written, and all the physical nodes of the cluster system When the average load value is equal to or less than the first predetermined threshold, the storage address to which the new data block and the fingerprint of the new data block are written is acquired from the current physical node.

  Certainly, in the storage policy of the new data block of the write address acquisition unit 97, the load information may not be specified again, the address is selected directly from the current physical node, and the write address acquisition unit 97 May be configured to obtain a storage address to which a new data block and a fingerprint of the new data block are written from the current physical node.

  For example, the writing unit 98 stores a new data block in the cache of the current physical node. When the data in the cache of the current physical node reaches the second predetermined threshold and the predetermined storage condition is satisfied, a new data block is stored. The data block and the fingerprint of the new data block may be written to the storage area corresponding to the selected storage address.

For the case where a virtual node is included in a cluster system, at least one virtual node is logically obtained through partitioning of the physical nodes of the cluster system, and correspondence between all virtual nodes and all physical nodes in the cluster system A relationship (VPT) is included in each physical node, and in an embodiment of the present invention, at least one virtual node is logically obtained through partitioning of each physical node of the cluster system,
Specifically, the specifying unit 92 specifies the first virtual node in the cluster system corresponding to each first sketch value of the n first sketch values according to the second algorithm, and determines the virtual node of the current physical node. It is configured to acquire the first physical node corresponding to the first virtual node by using the correspondence relationship between the node and the physical node as a query, and the acquired non-overlapping data block is used as a new data block Is done.

  Specifically, the update unit 96 specifies a second virtual node corresponding to each second sketch value according to the second algorithm by the second sketch value, and determines the second virtual node according to the correspondence between the virtual node and the physical node. Correspondence between each second sketch value and the storage address of the data block represented by each second sketch value after identifying the second physical node where the node is located and obtaining the storage address of the new data block Are stored in the second virtual node of the corresponding second physical node.

  When realizing the function of identifying the first virtual node in the cluster system corresponding to each of the n first sketch values according to the second algorithm, the identifying unit 92 corresponds to each first sketch value. In order to acquire the first virtual node in the cluster system, a modulo operation may be performed on the number of all virtual nodes of the cluster system with each first sketch value.

  In the case where the virtual node is included in the cluster system, when the data migration unit of the physical node of the cluster system is executed, the writing unit 98 further satisfies at least the physical node having data that needs to be migrated when the data migration state is satisfied. The virtual node is configured to integrally move one virtual node to the target physical node, and the update unit 98 further updates the correspondence relationship between the virtual node to be migrated of the current physical node and the physical node, and It is configured to notify that a correspondence relationship between a node to be migrated to another physical node and the physical node has been updated.

  The data migration condition may be that the data needs to be migrated to a physical node set in advance by the user or newly added during the capacity expansion of the physical node of the cluster system, and the load For balancing, data on a physical node with a high load is migrated to a physical node with a low load. The data migration condition is not limited in the embodiment of the present invention.

  The data processing apparatus of the cluster system may realize integral migration of virtual nodes in the physical node while the serial number of the virtual node to be migrated to the new physical node is not changed. The data storage address corresponding to the sketch value in the index table represents the location where the data is stored in the virtual node, and unless the virtual node identifier is changed, the data storage address does not need to be changed and the associated virtual node and physical Correspondence between nodes is changed, which simplifies the processing procedure during node expansion and greatly reduces the amount of computation.

  Referring to FIG. 10, the embodiment of the present invention further provides a data processing apparatus 100 in a cluster system. The apparatus may be set as a storage node of a cluster system, and includes a processor 101, a memory 102, a network controller 103, and a magnetic disk controller 104. The processor 101, primary memory 102, network controller 103, and magnetic disk controller 104 are Are connected via the bus 105.

  The storage controller 102 is configured to store the program 1021.

  The processor 101 is configured to execute the program 1021 of the primary memory 102.

  The processor 101 communicates with other nodes of the cluster system via the network controller 103, and communicates via a magnetic disk controller 104 with a storage device that can store data for a long time in the local node, such as a magnetic disk in FIG. The storage device controlled by the magnetic disk controller 104 is not limited to a magnetic disk, and may be a device formed by another storage medium.

  In a specific implementation, the memory 102 may be a primary memory such as a memory, and the specific implementation is not limited in the embodiments of the present invention.

  Specifically, the program 1021 may have a program code including a computer processing instruction.

  The processor 102 may be a central processing unit CPU or an application specific integrated circuit ASIC, or may be configured as one or more integrated circuits for implementing embodiments of the present invention.

  Referring to FIG. 9, the program 1021 may include the following.

  The dividing unit 90 is configured to divide the data stream received by the current physical node into m data blocks and obtain a fingerprint of each data block, where m is an integer of 1 or more.

  The sketch value acquisition unit 91 is configured to acquire n first sketch values representing a data stream according to the first algorithm, where n is an integer greater than or equal to 1 and less than or equal to m.

  For the first algorithm for obtaining the first sketch value, reference may be made to the description of the method described above, and details are not repeated here.

  The specifying unit 92 is configured to specify a first physical node in the cluster system corresponding to each first sketch value of the n first sketch values according to the second algorithm.

  There may be many ways for the identifying unit 92 to identify the first physical node. For example, route extraction and rounding is performed on the first sketch value until the value is less than m, and the final result is rounded to obtain the corresponding first physical node. In order to obtain the first physical node in the cluster system corresponding to each first sketch value, a modulo operation with each first sketch value may also be performed on the number of all physical nodes in the cluster system. Good.

  In the embodiment of the present invention, the identifying unit 92 obtains the first physical node in the cluster system corresponding to each first sketch value, and therefore each first node for each physical node in the cluster system. It is configured to perform a modulo operation with sketch values.

  The transmitter 93 is configured to transmit each first sketch value to the corresponding first physical node for a query.

  After receiving the query request transmitted by the transmitter 93, the other nodes in the cluster system query whether there is a storage address corresponding to the first sketch value according to the index table stored locally. In an embodiment of the present invention, each physical node or most physical nodes of the cluster system stores an index table between the sketch value and the storage address of the stored data block represented by the sketch value. The correspondence relationship is stored in the index table. The index table is stored in the physical node of the cluster system according to a predetermined policy, and the data block and fingerprint information corresponding to the data block are stored in storage areas indicated by different storage addresses.

  The receiving unit 94 receives at least one response message from the first physical node corresponding to the n first sketch values, and acquires the first storage address from the response message.

  The reception unit 94 obtains a storage address corresponding to the first sketch value from the response message, and collects a statistic regarding the number of hits of each storage address carried in the response message; And an address acquisition sub-unit 942 configured to select s storage addresses having the maximum number of hits as the first storage address according to the statistical result of the statistic collection sub-unit 941. Here, s is 1 or more.

  The query unit 95 compares the fingerprint of the data block stored in the storage area indicated by the first storage address with the fingerprint of each data block obtained through the division of the received data stream, and duplicate data Configured to query blocks. The acquired non-overlapping data block is used as a new data block.

  The update unit 96 acquires at least one second sketch value representing a new data block, specifies a second physical node corresponding to each second sketch value according to the second algorithm, using the second sketch value, and sets each second sketch value. A correspondence relationship between the sketch value and the storage address of the data block represented by each second sketch value is configured to be stored in the corresponding second physical node.

  The algorithm for acquiring the second sketch value representing the new data block by the update unit 96 is not limited here, but may be the same as the first algorithm for acquiring the first sketch value.

  The correspondence between each second sketch value and the storage address of the data block represented by each second sketch value is used as one index entry in the index table. In the embodiment of the present invention, the update unit 96 stores the index table in the physical node according to a predetermined policy, and there is no direct relationship between the data of the index table of a specific physical node and the data of the physical node. Instead, only the index table is stored in a specific physical node according to the set allocation policy. For the description of the index table, reference may be made to the description of the method described above, and details are not repeated here.

  With reference to the description of the method described above, in a specific implementation, there may be a plurality of methods by which the sketch value acquisition unit 91 acquires n sketch values representing a data stream according to the first algorithm. For example, the sketch value acquisition unit 91 may include a fingerprint grouping sub unit 911 and an acquisition sub unit 912, and the fingerprint grouping sub unit 911 obtains fingerprints of m data blocks of the data stream. It is configured to group into n first fingerprint groups, each first fingerprint group having a fingerprint of at least two data blocks.

  The acquisition sub-unit 912 is configured to acquire a first sketch value representing each fingerprint group in order to acquire n first sketch values.

  The acquisition sub-unit 912 may be configured to select the smallest fingerprint from each fingerprint group as the first sketch value representing the fingerprint group to which the smallest fingerprint belongs.

  The receiving unit 94 of the data processing apparatus of the cluster system further includes, for example, a case where the storage address fed back by the first physical node is not received after each first sketch value is transmitted to the corresponding first physical node for a query. The data block acquired through the division of the data stream as the non-overlapping data block may be used, and then the update unit 96 may be triggered.

  An apparatus provided by an embodiment of the present invention may be configured in a controller of a storage node of a cluster system and is configured to perform the data processing method described above. For a detailed description of the function of all units, reference may be made to the description of the method embodiment, and details will not be repeated here.

  When a duplicate data query is performed on the received data stream via the cluster system data processing apparatus provided by an embodiment of the present invention, the sampled sketch values are identified for the duplicate data query. The procedure of the duplicate data query transmitted to the physical node does not change with an increase in the number of nodes in the cluster system. Accordingly, the calculation amount of each node does not increase due to an increase in the number of nodes in the cluster system.

  When non-duplicate data is detected through the query, the query unit 95 regards the non-duplicate data as a new data block and stores the new data block in the single instance repository. For this reason, embodiments of the present invention further include:

  The write address acquisition unit 97 is configured to select a storage address to which a new data block and a fingerprint of the new data block are written according to a predetermined storage policy.

  The writing unit 98 is configured to write a new data block and a fingerprint of the new data block in a storage area corresponding to the selected storage address when a predetermined storage condition is satisfied.

There may be a plurality of storage policies for new data blocks in the write address acquisition unit 97. For example, the write address acquisition unit 97
A load information acquisition sub-unit 971 configured to acquire load information of each physical node of the cluster system;
According to the load information, a physical node to which data is migrated is selected, and a storage address to which a new data block and a fingerprint of the new data block are written is obtained from the physical node to which data is migrated An address selection sub-unit 972;
Have

  There may be a plurality of methods by which the address selection sub-unit 972 selects a storage address to which new data is written according to the load information. For example, if it is specified that the average load value of all the physical nodes of the cluster system has exceeded the first predetermined threshold according to the load information acquired by the load information acquisition sub unit 971, the node to which data is written A storage address selected from a physical node having a load value smaller than the load value and into which a new data block and a fingerprint of the new data block are written is obtained from the node to which the data is written, and all the physical nodes of the cluster system When the average load value is equal to or less than the first predetermined threshold, the storage address to which the new data block and the fingerprint of the new data block are written is acquired from the current physical node.

  Certainly, in the storage policy of the new data block of the write address acquisition unit 97, the load information may not be specified again, and the address is selected directly from the current physical node. Therefore, the write address acquisition unit 97 may be configured to acquire a storage address to which a new data block and a fingerprint of the new data block are written from the current physical node.

  For example, the writing unit 98 stores a new data block in the cache of the current physical node, and when the cache of the current physical node reaches the second predetermined threshold and satisfies a predetermined storage condition, The new data block fingerprint may be written to the storage area corresponding to the selected storage address.

  For the case where the virtual node is included in the cluster system, at least one virtual node is logically obtained through the division of the physical node of the cluster system, and the correspondence relationship (VPT) between all the virtual nodes and the physical node of the cluster system is obtained. ) Are included in each physical node, and in an embodiment of the present invention, at least one virtual node is logically obtained through partitioning of each physical node.

The specifying unit 92 corresponds to each first sketch value of the n first sketch values according to the second algorithm, specifies the first virtual node in the cluster system, and the correspondence between the virtual node and the physical node Is configured to acquire the first physical node corresponding to the first virtual node, and the acquired non-overlapping data block is used as a new data block,
The update unit 96 acquires at least one second sketch value representing a new data block, specifies a second virtual node corresponding to each second sketch value according to the second algorithm by the second sketch value, After specifying the second physical node where the second virtual node is arranged according to the correspondence with the physical node, and obtaining the storage address of the new data block, it is represented by each second sketch value and each second sketch value. The correspondence relationship between the storage addresses of the data blocks to be stored is stored in the second virtual node of the corresponding second physical node.

  When realizing the function of identifying the first virtual node in the cluster system corresponding to each sketch value of the n first sketch values according to the second algorithm, the identifying unit 92 corresponds to each first sketch value. In order to acquire the first virtual node in the cluster system, a modulo operation using each first sketch value may be performed on the number of all virtual nodes in the cluster system.

  In the case where the virtual node is included in the cluster system, when the data migration unit of the physical node of the cluster system is executed, the writing unit 98 further satisfies at least the physical node having data that needs to be migrated when the data migration state is satisfied. The virtual unit is configured to integrally move one virtual node to the target physical node, and the update unit 96 further updates the correspondence relationship between the virtual node to be migrated of the current physical node and the physical node, and It is configured to notify that a correspondence relationship between a node to be migrated to another physical node and the physical node has been updated.

  The data migration condition may be preset by the user, may be that data needs to be migrated to the newly added physical node during capacity expansion of the physical node of the cluster system, and For load balancing, data of a physical node with a high load may be transferred to a physical node with a low load. The data migration condition is not limited in the embodiment of the present invention.

  The cluster system's data processing device may realize integrated migration of the virtual node of the physical node while the serial number of the virtual node to be migrated to the new physical node does not change, corresponding to the sketch value of the index table The data storage address that represents the location where the data is stored in the virtual node. Therefore, as long as the virtual node identifier does not change, the data storage address does not need to be changed, only the correspondence between the virtual node and the physical node concerned is changed. Simplify and greatly reduce the amount of computation.

Embodiments of the present invention further provide a computer program for data processing, the computer program having a computer readable storage medium storing program code, the instructions included in the program code being:
Dividing the data stream received by the current physical node into m data blocks, obtaining a fingerprint of each data block, and obtaining n first sketch values representing the data stream according to a first algorithm Where m is an integer greater than or equal to 1, n is greater than or equal to 1 and is an integer less than or equal to m, and
Corresponding to each first sketch value of the n first sketch values according to a second algorithm, identifying a first physical node in the cluster system, and assigning each first sketch value to the corresponding first for querying Sending to the physical node;
Receiving at least one response message from a first physical node corresponding to the n first sketch values and obtaining a first storage address from the response message;
Comparing the fingerprint of the data block stored in the storage area indicated by the first storage address with the fingerprint of each data block obtained through the division of the data stream, and querying the duplicate data block A querying step in which the obtained non-overlapping data block is used as a new data block;
Obtaining at least one second sketch value representing the new data block, identifying a second physical node corresponding to each second sketch value according to the second algorithm by the second sketch value, and each second sketch value; Storing a corresponding relationship between the storage address of the data block represented by each second sketch value in the corresponding second physical node;
Used to execute

  A computer program for data processing provided by an embodiment of the present invention has a computer readable storage medium storing program code, and instructions included in the program code perform the method of the above-described method embodiment. For specific implementations, reference may be made to method examples, and details are not repeated here.

  For convenience and for simplicity of description, reference may be made to the corresponding processes of the method embodiments for the specific work processes of the systems, devices and units described above, and details are not repeated here.

  It should be understood that in the embodiments provided in this application, the disclosed system, apparatus and method may be implemented in other ways. For example, the unit division is merely a kind of logical function division, and there may be other division methods in the actual implementation. For example, multiple units or components may be combined or integrated into other systems, or some features may not be ignored or performed. Furthermore, the mutual coupling, direct coupling or communication connection shown or described may be realized via several communication interfaces. Indirect coupling or communication connections between devices or units may be realized electronically, mechanically or in other forms.

  A unit described as a separate part may or may not be physically separated, and a part shown as a unit may or may not be a physical unit. It may be arranged at one position, or may be distributed over a plurality of network units. Some or all of the units may be selected according to the actual needs to achieve the task of the means of the embodiments.

  Furthermore, the functional units of the embodiments of the present invention may be integrated into one processing unit, each unit may also be physically present alone, and two or more units may also be integrated into one unit. The

  When functions are implemented in the form of software functions and sold or used as independent products, these functions may be stored on a computer-readable storage medium. Based on this understanding, the technical means of the present invention or a part of the technical means contributing to the prior art substantially or partially may be realized in the form of a software product. The computer software product is stored in a storage medium and stored on a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the method steps described in the embodiments of the present invention. Contains a plurality of instructions for indicating. The above-described storage medium includes any medium that can store program codes, such as a USB flash disk, a removable hard disk, a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic disk, or an optical disk.

  The above descriptions are merely specific embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in the present invention shall fall within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (30)

A data processing method in a cluster system,
The current physical node divides the received data stream into m data blocks, obtains a fingerprint of each data block, and obtains n first sketch values representing the data stream according to a first algorithm. Obtaining m, wherein m is an integer greater than or equal to 1 and n is greater than or equal to 1 and less than or equal to m;
Corresponding to each first sketch value of the n first sketch values according to a second algorithm, identifying a first physical node in the cluster system, and assigning each first sketch value to the corresponding first for querying Sending to the physical node;
Receiving at least one response message from a first physical node corresponding to the n first sketch values and obtaining a first storage address from the response message;
The fingerprint of the data block stored in the storage area indicated by the first storage address is compared with the fingerprint of each data block obtained through the division of the data stream, and non-overlapping as a new data block Obtaining a data block;
Obtaining at least one second sketch value representing the new data block, identifying a second physical node corresponding to each second sketch value according to the second algorithm by the second sketch value, and each second sketch value; Storing a corresponding relationship between the storage address of the data block represented by each second sketch value in the corresponding second physical node;
Having a method. At least one virtual node is acquired and through the division of each physical node of the cluster system, correspondence between the virtual node and a physical node in the cluster system, included in each physical node,
Identifying a first physical node in the cluster system corresponding to each first sketch value of the n first sketch values according to the second algorithm;
Corresponding to each of the first sketch values of the n first sketch values according to the second algorithm, the first virtual node in the cluster system is specified, and the correspondence relationship between the virtual node and the physical node is queried. To obtain a first physical node corresponding to the first virtual node,
According to the second algorithm, a second physical node corresponding to each second sketch value is specified by the second sketch value, and between each second sketch value and the storage address of the data block represented by each second sketch value. In the corresponding second physical node,
A second virtual node corresponding to each second sketch value is specified by the second sketch value according to the second algorithm, and the second virtual node is arranged according to a correspondence relationship between the virtual node and the physical node. Identifying a physical node and storing a correspondence relationship between each second sketch value and a storage address of a data block represented by each second sketch value in a second virtual node of the corresponding second physical node; The method of claim 1, comprising: Identifying a first physical node in the cluster system corresponding to each first sketch value of the n first sketch values according to the second algorithm;
Performing a modulo operation with each first sketch value on the number of all virtual nodes in the cluster system to obtain the first virtual node in the cluster system corresponding to each first sketch value The method of claim 2 comprising: When the data migration condition is satisfied, the at least one virtual node of the physical node having data that needs to be migrated is integrally migrated to the target physical node;
The correspondence relationship between the virtual node and the physical node to be migrated of the current physical node is updated, and the correspondence relationship between the node to be migrated and the physical node is updated to another physical node of the cluster system. A step of notifying
The method according to claim 2, further comprising: Identifying a first physical node in the cluster system corresponding to each first sketch value of the n first sketch values according to the second algorithm;
Performing a modulo operation with each first sketch value on the number of all physical nodes in the cluster system to obtain the first physical node in the cluster system corresponding to each first sketch value The method of claim 1, comprising:   2. Obtaining at least one second sketch value representing the new data block comprises obtaining at least one second sketch value representing the new data block according to the first algorithm. The method according to any one of 5. Obtaining n first sketch values representing the data stream according to the first algorithm;
Grouping fingerprints of m data blocks of the data block into n first fingerprint groups, each first fingerprint group having a fingerprint of at least two data blocks Steps to
Obtaining a first sketch value representing each fingerprint group to obtain the n first sketch values;
The method according to claim 1, comprising: The step of acquiring the first storage address from the response message includes:
A storage address corresponding to the first sketch value is acquired from the response message, statistics regarding the number of hits of each storage address conveyed in the response message are collected, and s having the maximum number of hits as the first storage address 6. The method according to any one of claims 1 to 5, comprising the step of selecting storage addresses, wherein s is one or more.   When a predetermined storage condition is satisfied, a storage address to which the new data block and the fingerprint of the new data block are written is selected according to a predetermined storage policy, and the finger of the new data block and the new data block are selected. 6. The method according to claim 1, further comprising the step of writing a print in a storage area indicated by the selected storage address. Selecting a storage address to which the new data block and the fingerprint of the new data block are written according to the predetermined storage policy;
Obtaining load information of physical nodes of the cluster system, selecting a physical node to which data is migrated according to the load information, and selecting the new data block and the new data block from the physical node to which the data is migrated The method according to claim 9, further comprising obtaining a storage address where the fingerprint is written. Obtaining load information of physical nodes of the cluster system, selecting a physical node to which data is migrated according to the load information, and selecting the new data block and the new data block from the physical node to which the data is migrated The step of obtaining the storage address where the fingerprint is written is as follows:
When the load information of the physical nodes of the cluster system is acquired and the average load value of all the physical nodes of the cluster system exceeds a first predetermined threshold, data is transferred from the physical node having a load value smaller than the average load value. Selecting a node to be migrated and obtaining a storage address to which the new data block and the fingerprint of the new data block are written from the node to which the data is migrated;
Storage in which the new data block and the fingerprint of the new data block are written from the current physical node when the average load value of all the physical nodes of the cluster system is less than or equal to the first predetermined threshold Obtaining an address;
The method of claim 10, comprising: Selecting a storage address to which the new data block and the fingerprint of the new data block are written according to the predetermined storage policy;
10. The method of claim 9, comprising obtaining a storage address to which the new data block and the fingerprint of the new data block are written from the current physical node. Satisfying the predetermined storage condition includes storing the new data block in a cache of the current physical node;
The method of claim 9, wherein the predetermined storage condition is satisfied when a number of data blocks in the cache of the current physical node reaches a second predetermined threshold.   Obtaining a first sketch value representing each fingerprint group comprises selecting a smallest fingerprint from each fingerprint group as a first sketch value representing the fingerprint group to which the smallest fingerprint belongs. The method of claim 7. The fingerprint of the data block stored in the storage area indicated by the first storage address is compared with the fingerprint of each data block obtained through the division of the data stream, and non-overlapping as a new data block The step to get the data block is
Load the fingerprint of the data block in the storage area pointed to by the first storage address into the current physical node and perform comparison with the fingerprint of the m data blocks obtained through the division of the data stream The method according to claim 1, further comprising: obtaining the non-overlapping data block as the new data block. The fingerprint of the data block stored in the storage area indicated by the first storage address is compared with the fingerprint of each data block obtained through the division of the data stream, and non-overlapping as a new data block The step to get the data block is
Transmitting a query instruction to the physical node indicated by the first storage address and receiving a query result returned by the physical node indicated by the first storage address;
15. A method according to any one of the preceding claims, wherein a fingerprint of m data blocks obtained via the division of the data stream is carried by the query indication.   After sending each first sketch value to the corresponding first physical node for a query, the method further does not receive a storage address returned by the first physical node corresponding to the n first sketch values. 15. A method as claimed in any preceding claim, comprising using data blocks obtained via division of the data stream as non-overlapping data blocks. A data processing apparatus in a cluster system,
A dividing unit that divides a data stream received by a current physical node into m data blocks and obtains a fingerprint of each data block, wherein m is an integer greater than or equal to 1,
A sketch value acquisition unit for acquiring n first sketch values representing the data stream according to a first algorithm, wherein n is an integer not less than 1 and not more than m;
A specifying unit for specifying a first physical node in the cluster system corresponding to each of the first sketch values of the n first sketch values according to a second algorithm;
A transmitter for transmitting each first sketch value to the corresponding first physical node for a query;
A receiving unit that receives at least one response message from a first physical node corresponding to the n first sketch values, and acquires a first storage address from the response message;
The fingerprint of the data block stored in the storage area indicated by the first storage address is compared with the fingerprint of each data block obtained through the division of the data stream, and non-overlapping as a new data block A query part to obtain a data block;
Obtaining at least one second sketch value representing the new data block, identifying a second physical node corresponding to each second sketch value according to the second algorithm by the second sketch value, and each second sketch value; And an update unit that stores a corresponding relationship between the storage address of the data block represented by each second sketch value in the corresponding second physical node;
Having a device. A write address acquisition unit configured to select a storage address to which the new data block and the fingerprint of the new data block are written according to a predetermined storage policy;
A write unit configured to write the new data block and the fingerprint of the new data block to a storage area corresponding to the selected storage address when a predetermined storage condition is satisfied;
The apparatus of claim 18, further comprising: At least one virtual node is acquired and through the division of each physical node of the cluster system, correspondence between the virtual node and a physical node in the cluster system, included in each physical node,
The specifying unit corresponds to each first sketch value of the n first sketch values according to the second algorithm, specifies a first virtual node in the cluster system, and between the virtual node and the physical node. A first physical node corresponding to the first virtual node is obtained by making the correspondence relationship of
The non-overlapping data block acquired by the query unit is used as the new data block,
The update unit acquires at least one second sketch value representing the new data block, specifies a second virtual node corresponding to each second sketch value by the second sketch value according to the second algorithm, A second physical node on which the second virtual node is arranged according to a correspondence relationship between the virtual node and the physical node, and each second sketch value and a storage address of a data block represented by each second sketch value; 20. The apparatus of claim 19, wherein the apparatus is configured to store a correspondence relationship between a second virtual node of the corresponding second physical node. The update unit is further configured to integrally migrate at least one virtual node of physical nodes having data that needs to be migrated to the target physical node when a data migration condition is satisfied,
The update unit further updates a correspondence relationship between the virtual node to be migrated of the current physical node and the physical node, and the node between the node to be migrated to another physical node of the cluster system and the physical node. 21. The apparatus of claim 20, wherein the apparatus is configured to notify that a correspondence relationship has been updated.   The specifying unit corresponds to each first sketch value, and acquires the first physical node in the cluster system, so that the number of all physical nodes in the cluster system is modulo by the first sketch value. The apparatus of claim 18, wherein the apparatus is configured to perform an operation. The sketch value acquisition unit
A fingerprint grouping sub-unit configured to group fingerprints of m data blocks of the data stream into n first fingerprint groups, each first fingerprint group comprising at least two A fingerprint grouping sub-part having a fingerprint of the data block;
An acquisition sub-unit configured to acquire a first sketch value representing each fingerprint group to acquire the n first sketch values;
The apparatus according to claim 18, comprising: The receiver is
A statistics collection sub-unit configured to obtain a storage address corresponding to the first sketch value from the response message and collect statistics about the number of hits of each storage address carried in the response message;
An address acquisition sub-unit configured to select s storage addresses having the maximum number of hits as the first storage address according to a statistical result of the statistic collection sub-unit, wherein s is 1 or more; An acquisition sub-part,
The apparatus according to claim 18, comprising: The write address acquisition unit
A load information acquisition sub-unit configured to acquire load information of physical nodes of the cluster system;
In accordance with the load information acquired by the load information acquisition sub-unit, the physical node to which data is migrated is selected, and the new data block and the fingerprint of the new data block are selected from the physical node to which the data is migrated. An address selection sub-part configured to obtain a storage address to which is written;
The apparatus according to claim 19, comprising:   The write address acquisition unit is configured to acquire, from the current physical node, a storage address to which the new data block and the fingerprint of the new data block are written. The device according to item. The update unit is configured configured to store the new data block in the cache of the current physical node, the number of cache data blocks in the current physical node reaches Then the second predetermined threshold value, said predetermined storage conditions Ru is satisfied, claim 25 apparatus according.   The receiving unit further receives a storage address returned by the first physical node corresponding to the n first sketch values after transmitting each first sketch value to the corresponding first physical node for a query. 28. An apparatus as claimed in any one of claims 18 to 27, wherein if not, the apparatus is configured to utilize a data block obtained via division of the data stream as a non-overlapping data block. A data processing apparatus in a cluster system,
A processor;
Memory,
With bus,
Have
The processor and the memory communicate with each other via the bus;
The memory is configured to store a program;
The apparatus configured to execute the program of the memory and to execute the method according to claim 1.   A computer program for causing a computer to execute the method according to claim 1.

Images