JP2017514217A - Data processing method and apparatus - Google Patents

Abstract

Data processing technology is provided, and the data processing technology is applied to the partition management device. The partition management device stores a partition view, the partition view records an association between the ID of the current partition and the address of the storage disk, and the total number of current partitions may be less than the total number of confirmed partitions . By using the technique, data transfer can be performed on key-value data using the current partition, thereby reducing the complexity of the partition view.

Description

  The present application relates to data processing technology, and in particular, to a data processing method and apparatus.

  As society evolves, data that continues to increase in volume needs to be stored and managed, and is also referred to as massive data. When managing very large data using traditional centralized storage, it is almost impossible to provide efficient read / write processing, and it is almost impossible to satisfy excellent scalability and high availability.

  Against this background, a storage system composed of a plurality of physical storage nodes has emerged in which each storage node can provide a storage area. This storage system is called distributed storage. There is a distributed storage system called key-value storage. In key-value storage, stored data (or data fragment) is called value, and each part of data has a unique identifier in the entire storage system, and this identifier is a key (key). ). There is a one-to-one correspondence between keys and values.

  A key and a value corresponding to the key are generally referred to as a key value, abbreviated as K-V. Each key value is stored on a storage disk of the storage system. In the distributed hash table (DHT) technique, for a specific key value, a storage disk in which the key value is stored can be determined using a mapping rule. This mapping rule is based on a hash value generated by performing a hash operation on the key, where each hash value corresponds to one storage disk. The value belongs to one partition, and the partition corresponds to the storage disk. Based on this method, if the calculated hash values of two different keys are the same, the key values corresponding to the two keys are stored on the same storage disk. The association between a partition and a storage disk is called a partition view.

  In the prior art, the Hash value calculated according to the key using the DHT technique is included in an integer range of [0, 2 ^ 32-1], for example. During system initialization, segmentation is performed over this large integer range, with each segment being equal or approximately equal in size. In this case, one segment is a partition, and the number of hash values in the partition is basically the same. If the number of storage disks in the storage disk cluster is relatively small, each storage disk will have more partitions than necessary, causing the partition view to become overly complex and consequently follow the partition view Data packet transfer becomes inefficient. A specific example is as follows.

  Assuming that there is a cluster that supports up to 25,000 storage disks, in the case of the maximum number of storage disks, each storage disk has approximately 100 partitions. That is, the entire cluster has a total of 2,500,000 partitions. Assume that the information in each partition occupies a 4-bit storage area. The information in these partitions will occupy a total of 10MB of storage space, and the information in the partition view will be larger than 10MB.

  When using the partition view, a large amount of system resources are occupied.

  The present invention provides a data processing method and apparatus that can reduce the occupation of system resources when processing key-value data.

  According to a first aspect, the present invention provides a data processing method applied to a partition management device, the partition management device stores a partition view, and the partition view includes an ID of a current partition and a storage disk. The method is a step of obtaining a key in the key-value data and calculating an ID of a confirmed partition corresponding to the key-value data according to the key, the method comprising: Is a step that includes a value and a key that uniquely corresponds to the value, and a step of calculating an ID of the current partition corresponding to the ID of the confirmed partition, wherein each current partition ID includes a plurality of confirmed partitions. The step corresponding to the ID and the current To get the address of the storage disk corresponding to the partition ID, search the partition view and generate the key value packet by using the storage disk address as the destination address, The key value packet includes a step of carrying key value data.

  According to a second aspect, the present invention provides a data processing apparatus, wherein the apparatus is a storage module configured to store a partition view, wherein the partition view includes an ID of a current partition. The storage module that records the association between the address of the storage disk and the key in the key-value data is obtained, and the ID of the confirmed partition corresponding to the key-value data is calculated according to the key In the definite partition calculation module, the key-value data includes the deterministic partition calculation module including the key that uniquely corresponds to the value and the value, and the ID of the current partition corresponding to the deterministic partition ID is calculated. Configured current partition calculation module The ID of each current partition is stored by the storage module to obtain the current partition calculation module corresponding to the IDs of multiple confirmed partitions and the storage disk address corresponding to the ID of the current partition. Configured to search for a partitioned partition view and configured to generate a key-value packet by using the address of the storage disk as the destination address and send the key-value packet to the storage disk The key-value packet is a transmission module that carries key-value data.

  According to a third aspect, the present invention provides a data processing device, the data processing device being configured to store a partition view, wherein the partition view is an ID of a current partition. A computer-readable medium configured to store a computer program, a memory configured to provide an external interface, a memory that records an association between a storage disk address, and a memory; Obtaining a key in the key-value data by operating a program and being connected to an interface and a computer-readable medium, and calculating an ID of a confirmed partition corresponding to the key-value data according to the key, Key value data is A step including a value and a key uniquely corresponding to the value, and a step of calculating an ID of the current partition corresponding to the ID of the confirmed partition, wherein each current partition ID is set to an ID of a plurality of confirmed partitions To find the corresponding step and the address of the storage disk corresponding to the ID of the current partition, generate a key-value packet by searching the partition view and using the storage disk address as the destination address And sending the key value packet to the storage disk via the interface, wherein the key value packet comprises a processor configured to carry the step of carrying the key value data.

  According to a fourth aspect, the present invention provides a partition management method executed by a controller, the controller performs partition management for storage disks in the cluster, and the cluster stores a plurality of storage disks. The method includes the step of obtaining the number M of current storage disks in the cluster and the total number T of currently existing partitions in the cluster when detecting that N new storage disks are about to be added to the cluster. M, N, and T are all natural numbers, and whether the mathematical relationship between the step and the total number T of current partitions and the total number M + N of storage disks satisfies the first preset condition And if the relationship satisfies the first preset condition, This step divides at least one current partition so that the total number of partitions acquired later is S, and allocates the partitions acquired after the partition to M + N storage disks, and the total number of partitions acquired after the division The mathematical relationship between S and the total number of storage disks M + N satisfies the second preset condition, the total number of partitions obtained after the split is not greater than the total number L of committed partitions supported by the cluster, and L and Both S include steps that are natural numbers greater than one.

  In the first embodiment of the fourth mode, the process of updating the partition view is further executed. Here, the partition view records the association between the current partition and the IP disk.

  According to a fifth aspect, the present invention provides a partition management device configured to perform partition management for storage disks in a cluster, wherein the cluster includes a plurality of storage disks, and the device Is configured to obtain the number M of current storage disks in the cluster and the total number T of currently existing partitions in the cluster upon detecting that N new storage disks are about to be added to the cluster, Storage disk detection module, where M, N, and T are all natural numbers, and the mathematical relationship between the storage disk detection module and the total number T of current partitions and the total number M + N of storage disks is Configured to determine if the preset condition of 1 is met, the first If the relationship between the preset condition determination module and the first preset condition is satisfied, divide at least one current partition so that the total number of partitions acquired after splitting is S, and the partition acquired after splitting is M + A partitioning module configured to allocate to N storage disks, the mathematical relationship between the total number of partitions S obtained after the split and the total number of storage disks M + N is the second preset condition And the total number of partitions obtained after the partitioning is not larger than the total number L of deterministic partitions supported by the cluster, and both L and S are provided with a partitioning module that is a natural number larger than 1.

  In a first embodiment of the fifth aspect, the partitioning module is further configured to perform a process of updating the partition view, wherein the partition view is between the current partition and the IP disk. Record the correspondence.

  According to a sixth aspect, the present invention provides a partition management device that is connected to a cluster and configured to perform partition management on storage disks in the cluster. The partition management device is a memory configured to store a partition view, wherein the partition view records an association between the ID of the current partition and the address of the storage disk, A computer readable medium configured to store an interface and a computer program, and connected to the memory, the interface and the computer readable medium; By using the interface to detect that N new storage disks are about to be added to the cluster, the number M of current storage disks in the cluster and the total number T of currently existing partitions in the cluster are calculated. Steps to acquire, where M, N, and T are all natural numbers, and the mathematical relationship between the step and the total number of current partitions T and the total number of storage disks M + N is the first preset condition And if the relationship satisfies the first preset condition, divide at least one current partition so that the total number of partitions obtained after splitting is S, and after splitting Allocating the acquired partition to M + N storage disks, The mathematical relationship between the total number of partitions S acquired after splitting and the total number of storage disks M + N satisfies the second preset condition, and the total number of partitions acquired after splitting is the total number of confirmed partitions supported by the cluster Both L and S comprise a processor configured to perform a step that is not greater than L and both L and S are natural numbers greater than one.

  In the first embodiment of the sixth aspect, the processor further executes a process of updating the partition view, where the partition view records the association between the current partition and the IP disk.

  By using the solution of the present invention, the partition management device may use the current partition to perform key-value data transfer. Since the number of current partitions is less than that of the deterministic partition, resource consumption is reduced compared to the prior art solution used by the deterministic partition to perform the transfer.

1 is a topology diagram of an embodiment of a storage system according to the present invention. FIG. 6 is a flowchart of an embodiment of a partition management method according to the present invention; FIG. 4 is a structural diagram of an embodiment of a controller according to the present invention. 1 is a schematic diagram of an embodiment of a partition management device according to the present invention. 4 is a flowchart of an embodiment of a data processing method according to the present invention; FIG. 6 is a structural diagram of an embodiment of a data processing device according to the present invention; 1 is a schematic diagram of an embodiment of a data processing apparatus according to the present invention.

  The technical solutions in the embodiments of the present invention will be described clearly below with reference to the accompanying drawings in the embodiments of the present invention. It will be apparent that the embodiments described are only a part rather than all of the embodiments of the present invention. All other embodiments obtained based on the embodiments of the present invention shall fall within the protection scope of the present invention.

  As shown in FIG. 1, FIG. 1 is a topology diagram of a storage system according to an embodiment of the present invention. The application server 11 is connected to the management server cluster 12, the management server cluster 12 is connected to the switch cluster 13, the switch cluster 13 is connected to the IP disk cluster 14, and the controller 15 is managed Connected to server cluster 12 and IP disk cluster 14. The management server cluster 12 includes at least one management server 121, and the IP disk cluster 14 includes at least one IP disk 141.

  The application server 1 transmits a data read command or a data write command. The management server 121 is, for example, a distributed object pool (DOP), and provides an interface of the object to the application server 11. Here, the object may be very large, for example, a basic unit. May be GB.

  If the object is too large, the management server 121 may divide the object into small segments. For example, the management server 121 may divide the object into segments each having a size of 1 MB, where each segment is a value and each value has a tag called a key. The management server 121 can perform a hash operation on the key and perform matching between the calculated hash value and the partition. In addition, the management server 121 may further store a partition view, where the partition view may be an association between the ID of the current partition and the IP disk, in particular the correspondence between the IP disk and the partition. Record the date. Therefore, the management server can search for an IP disk corresponding to each key value. The IP disk to be found is referred to as a target disk, and the management server 121 can generate an IP packet by using the address of the target IP disk as a destination address and transmit the IP packet to the switch cluster 13. For example, the contents recorded in the partition view are shown in Table 1, where i is a natural number and m is a natural number greater than 1.

  The switch cluster 13 includes at least one switch, and is used for exchanging data between the management server 121 and the IP disk 141 during data reading or data writing. The process of retrieving the partition view is also left to the switches in the switch cluster 13 and may be performed by the switches in the switch cluster 13, that is, the switches in the switch cluster 13 store the partition view. In this scenario, when data is transmitted to the switch cluster 3, the management server 121 may use a new type of packet that can be referred to as a key-value packet without using an IP packet. The difference between the key value packet and the IP packet is that the destination address of the key value packet is the partition number obtained by calculation using the key, and the packet type field distinguishes between the IP packet and the key value packet. The other part of the key-value packet is the same as that of the IP packet. The switch searches the mapping table between the partition number and the IP disk address, where the mapping table is stored by the switch, replaces the destination address with the IP disk address corresponding to the partition, and the key-value packet The packet type is modified to convert the IP packet into an IP packet, and then transferred to the corresponding IP disk for storing the IP packet.

  In the management server 121, the method of recording the association between the partition and the IP disk may be to record the association between the partition number (also referred to as partition ID) and the IP disk address. . In order to facilitate understanding, if another representation method of association is used, the partition belongs to the IP disk, and if another representation method is used, the IP disk owns the partition.

  A partition is a logical concept. A partition does not own a storage area, and it is an IP disk that actually stores data. However, each partition has a corresponding IP disk, and during the data storage process, the IP disk corresponding to the partition can be learned by searching the partition view to perform the actual storage. From the user's point of view, the data appears to be stored in the partition, therefore sometimes we do the process of storing the data in the IP disk corresponding to the partition as storing the data in the partition Sometimes it points.

  The controller 15 is configured to calculate the total number of partitions, obtain information about the IP disk, establish a correspondence between the IP disk and the partition, and update the correspondence in each management server as appropriate. The association is also referred to as a partition view.

  In the prior art, since the total number of partitions is constant, the number of partitions corresponding to each IP disk decreases as the number of IP disks increases. Therefore, when the number of IP disks is relatively small, each IP disk corresponds to more partitions than necessary. For example, in the example in the background art, if there are 25,000 storage disks, each storage disk has 100 partitions, which is appropriate, however, in the case where there are 75 storage disks, Each storage disk has 33,333 partitions, which is inappropriate because it can cause at least the following problems:

  (1) Since the partition view has 75 × 33,333 = 2,499,975 associations that approximate the number of associations in the case where 25,000 storage disks exist, the partition view becomes excessively complicated. When transferring the IP packet using the partition view, the management server needs to read the association in the partition view. When the number of IP disks is small, it is necessary to read such a large amount of correspondence, resulting in low transfer efficiency. In addition, if the partition view is made known, the controller needs to make the partition view known to each management server, which occupies a relatively large bandwidth. Accordingly, the management server occupies more processing resources than necessary and bandwidth resources more than necessary in the storage system.

  (2) To improve data reliability, each partition has a duplicate partition on another IP disk, where the partition and its duplicate partition are located on different IP disks, The same data is stored. This case is called a plurality of duplicates. Assume that partition B and partition C are duplicate partitions of partition A. When a packet is stored on the IP disk where partition A is located, the packet is stored simultaneously on the IP disk where partition B is located and on the IP disk where partition C is located Yes.

  Assume that a failure occurs on an IP disk in multiple duplicate scenarios. Data in a failed storage disk needs to be recovered by using duplicates to ensure data reliability. Specifically, a faulty IP disk is searched for a duplicate partition of the partitions, and data in the duplicate partition is copied. The copied data is stored on an IP disk that has not failed. If there are more partitions than necessary in each IP disk, the number of duplicate partitions for these partitions will also be very large, and these duplicate partitions may be on a large number of other IP disks. As a result, if one IP disk fails, a large number of other IP disks will also be involved in data recovery, affecting the performance of all these IP disks involved in data recovery during data recovery. Effect.

  (3) In multiple duplicate scenarios, if each IP disk has an excessive number of partitions, duplicate IP disk partitions will be scattered over a large number of IP disks. Become. Since the number of IP disks storing duplicates is relatively large, the probability that a failure will occur simultaneously on multiple IP disks increases, thereby reducing data reliability.

  (4) This embodiment of the present invention introduces the concept of parent partition and deterministic partition, where the total number of deterministic partitions is constant, which is similar to partitioning in the prior art . A parent partition can be divided into multiple subpartitions, and if a subpartition can be divided into new subpartitions, the subpartition is the parent partition of the next stage subpartition. As the quantity of IP disks in the cluster increases, the splitting process can continue until a deterministic partition is obtained by splitting. In this case, the number of partitions owned by the entire cluster reaches the total number of confirmed partitions.

  Note that in the prior art, the number of partitions visible to the user is the same as the number of partitions managed within the storage system. The number of partitions is constant no matter how many IP disk clusters are owned, and at the same time, what is visible to the user in this embodiment of the invention is always the number of deterministic partitions. However, before the parent partition is divided into fixed partitions, the storage system is managed using the parent partition. That is, both partition view and packet forwarding are handled using the parent partition. In the embodiment of the present invention, unless otherwise specified, the partition refers to a partition managed by the storage system.

  If there is a relatively small amount of IP disks in the cluster, an appropriate number of parent partitions are allocated to each IP disk, and the merged partition quantity for each IP disk is specified by the administrator or automatically by the system. May be set and is not limited by the number of committed partitions. For example, if a cluster has 75 IP disks, each IP disk corresponds to 100 parent partitions, and only 75 x 100 = 7,500 associations are displayed in the partition view. Will exist. If the cluster has 150 IP disks and each parent partition is divided into two subpartitions, each IP disk still supports 100 partitions, and each partition Assuming you own one duplicate, if a failure occurs on an IP disk, the number of other affected IP disks will be limited to 100 or less. In other words, the present invention can implement partition quantity customization so that the number of partitions owned by each IP disk can be controlled. Based on the method provided in this embodiment of the present invention, if the controller sends a partition view to each management server in the management server cluster, less bandwidth will be occupied. In addition, after the number of matching entries is reduced, the management server can query the address of the storage disk corresponding to the key faster.

  When the application server 1 reads data or needs to write data, the data sequentially reaches the IP disk through the management server 21 and the switch cluster 3.

  The IP disk uses a magnetic disk or flash memory as a storage medium, provides a key-value interface in the software, provides an Ethernet (registered trademark) interface in hardware, and receives via the Ethernet (registered trademark) interface An IP packet can be acquired by expanding the Ethernet (registered trademark) frame. The IP disk is, for example, the Seagate Technology Kinetic product.

  In this embodiment of the present invention, the memory for storing the key value data is generically called a storage disk. In addition to the IP disk, in another embodiment, the storage disk may also use another storage device that supports a key-value interface and use a non-Ethernet interface in hardware. The storage medium used by the storage disk may be a hard disk or flash memory.

  2 introduces the partition management method according to the embodiment of the present invention, and increases the number of partitions to meet the requirements of newly added IP disks when the number of IP disks in the IP disk cluster increases Explains how to do. The method is performed by a controller, and the method includes the following steps.

  Step 21: A fixed partition quantity L, which is a natural number, is determined according to the maximum number of IP disks supported by the cluster, and the initial partition quantity owned by each IP disk is determined according to the initial IP disk quantity. This step is a pre-step that is performed when the controller is initialized and is optional. In addition, each initial partition corresponds to one storage disk, and this association is recorded by using the partition view.

  After this step is executed, the controller records the initial IP disk quantity as the current IP disk quantity M and the initial partition quantity owned by each IP disk as the current partition quantity P, where The total number T of currently existing partitions in the cluster is T = M · P. The controller further records the quantity L of the confirmed partition. Thereafter, when the number of IP disks in the cluster changes or the number of partitions of each IP disk changes, the number M of current IP disks and the number P of current partitions are updated. In the embodiment of the present invention, the symbol “•” represents a product.

  The number of deterministic partitions is constant, and deterministic partitions cannot be divided and can be perceived by the user. An initial partition is generally a parent partition, and a parent partition is a partition that can be split to generate a next stage parent partition or a deterministic partition by splitting. The parent partition is used by the storage system and the user cannot perceive the presence of the parent partition. In this embodiment of the present invention, the current partition refers to a partition that is currently used by the management server and is made known to the management server by the controller. When partition partitioning is executed, the current partition is a partition acquired after partition partitioning. The current partition can be a parent partition or can include a deterministic partition.

  The value of L can be set by the user or automatically allocated by the system, depending on both the maximum number of IP disks and the number of partitions on each IP disk when the number of IP disks is the maximum Generally determined, that is, L = maximum number of IP disks × number of partitions of each IP disk. Assume that owning X partitions is the desired value for each IP disk. For example, if each IP disk owns 100 partitions, that is, the user thinks that X = 100 is a relatively appropriate value, the maximum number of IP disks that an IP disk cluster can support is As a result, L = 10,000 × 100 = 1,000,000.

  Similarly, the initial partition quantity can be set by the user or automatically assigned by the system. One optional scheme is that in the initial situation, assuming that the number of IP disks M is M = 75 and 100 partitions are allocated to each IP disk, the result is that the total number of initial partitions = 75 × 100 = 7,500. By using the method of the present invention, in the initial stage, the number of partitions owned by each IP disk in the storage system may be freely set, where the number of partitions meets the user's requirements. A partition may be set to a value that does not occupy more resources than necessary, such as storage resources, computational resources, and bandwidth resources.

  Each initial partition corresponds to a confirmed partition. Each initial partition has an ID, and the ID number is an integer of 0 or more. Each confirmed partition has an ID, and the number is an integer of 0 or more. The method for obtaining the deterministic partition corresponding to the initial partition is to perform a modulo operation on the total number of initial partitions using the deterministic partition ID, and the remainder value is the initial value corresponding to the deterministic partition. Indicates the partition ID. In this case, the initial partition is the current partition.

  Step 22: Upon detecting that N new IP disks are going to be added to the IP disk cluster, obtain the number M of current storage disks in the cluster and the total number T of partitions currently present in the cluster, where , M, N, and T are all natural numbers.

  The controller is connected to the IP disk cluster, so the new IP disk that is about to be added to the cluster can be detected by the controller. At present, there are already M IP disks, and the total number of current partitions is MP. The N IP disks are physically connected to the cluster and can be detected by the controller, but cannot store data because the partition is not allocated to the IP disk.

  Note that “current” refers to the point in time at which this step is about to be performed. In this embodiment, no IP disks are added after the controller is initialized, so the number of current IP disks is M. In another embodiment, if the number of IP disks in the IP disk cluster has changed before this step is performed, the number of current IP disks is not M. If the partition was partitioned before this step was executed, the number of initial partitions currently owned by each IP disk is greater than P. T partitions are allocated approximately evenly across M IP disks.

  Step 23: Determine whether the current partition quantity meets the common requirements of the current IP disk and the newly added IP disk, i.e. between the total number of current partitions T and the total number of storage disks M + N Determine whether the mathematical relation of satisfies the first preset condition. Specifically, the determination may be performed by a method of comparing the formula M · P / (M + N) and the first threshold, where the first threshold is a natural number and the first threshold May be preset by the controller. When this step is executed for the first time after the controller is initialized, T = M · P. Optionally, the first threshold described above may be an integer greater than 10 and less than 20, such as 16, 17, 18, or 19.

  In one determination method, when T / (M + N) is less than the first threshold, the first preset condition is satisfied, and division is necessary. If the number of partitions owned by each IP disk is less than the partitioning threshold (or can be set below the partitioning threshold), the number of partitions owned by each IP disk will be too small without performing partitioning. Therefore, the total number of partitions needs to be increased by partitioning, so that step 24 is executed. If the number of partitions owned by each IP disk is not less than the division threshold, step 25 is executed.

  Another determination method is that if the average number of partitions owned by each IP disk is greater than the threshold (or can be set to be higher than the threshold) after the first partitioning, each IP disk can be divided. Indicates that the number of partitions owned by is excessively large, so that step 25 is executed. If the average number of partitions owned by each IP disk is not greater than the threshold, step 24 is executed. In addition, the two decision methods may also be combined to make a decision, and the solution with the highest satisfaction with the service is selected. All of the following embodiments illustrate the present invention by using the first determination method as an example.

  Step 24: Divide at least one current partition that can be divided once or multiple times until the number of partitions obtained after the division satisfies the requirement, and then execute step 26. The fact that the number of partitions acquired after splitting satisfies the requirement means that the mathematical relationship between the total number S of partitions acquired after splitting and the total number of storage disks M + N satisfies the second preset condition and is acquired after splitting. It may be that the total number of partitions made is not larger than the total number L of confirmed partitions. Dividing a plurality of times refers to dividing a partition generated by the division a plurality of times.

  The mathematical relationship between the total number of partitions obtained after splitting S and the total number of storage disks M + N satisfies the second preset condition, in particular, S / (M + N) is greater than or equal to the second threshold And the second threshold may be a natural number. The second threshold may be a natural number greater than 25 and less than 50, such as 25, 26, 27, 48, or 49, for example.

  There can be various schemes for determining whether the quantity of partitions obtained after splitting meets the requirement. For example, when the mathematical relationship between the total number S of partitions acquired after the division and the total number M + N of the storage disks satisfies the second preset condition, the division ends. Specifically, for example, when the average number of partitions owned by each IP disk after the division is equal to or greater than the second threshold, it is considered that the second preset condition is satisfied, and the division ends. Alternatively, if the average number of partitions owned by each divided IP disk satisfies the preset threshold range, it is considered that the requirement is satisfied.

When each partition is divided into a plurality of subpartitions in each division, each partition is divided into the same number of subpartitions. A magnification that changes the number of partitions from the first division to the end of the division is referred to as a division coefficient. Assume that each parent partition is split into two subpartitions in each split. As a result, when the division after the first division is completed, the decimation factor is 2, when the split after the second division is completed, the decimation factor is 2 ^2.

  If dividing at least one current partition indicates dividing all current partitions, the total number of partitions obtained after the division is S = T × the division coefficient. When dividing at least one current partition indicates dividing some current partitions, the total number of partitions obtained after the division is S <T × division coefficient. The former division method has a higher division speed, and the latter division method has a better adjustment function of the total number of partitions obtained after the division. For convenience of explanation, this embodiment of the present invention will be described using the former division method.

  An example of the latter division method is shown below. Assuming that the total number of deterministic partitions is 1,000, the total number of current partitions is 512, and the division factor is 2, dividing each partition will result in 1,024 partitions exceeding the total number of deterministic partitions after being divided Will be acquired. The total number of current partitions is not allowed to be greater than the total number of committed partitions. To avoid such a case, only 488 current partitions can be split, i.e. 488 + 512 = 1,000, i.e. the total number of current partitions obtained after splitting has just reached 1,000, and It will not be larger than the total number. In addition, if the user thinks that 1,000 current partitions are an excessively large quantity, for example, if the user thinks that it is most appropriate that the total number of current partitions obtained after splitting reaches 800, then splitting The 288 partitions of the 512 current partitions can be divided according to the division factor 2 so that the total number of current partitions acquired later just reaches 800. Using the latter partitioning method, it may be that only one partition is split each time, and when the partitioning factor is 2, the difference in the total number of current partitions before and after the partitioning is 1, so The granularity for partitioning is minimal.

  In addition, the division coefficient may be changed based on the above-described embodiment. For example, 2 is used as the division coefficient during the first division, but the division coefficient is changed to 5 during the second division. This can also achieve the purpose of making the value of the total number of partitions obtained after the division more adjustable.

  There is an association between the current partition and the confirmed partition, and this association can be stored in the controller or can be made known to each management server by the controller. Each current partition has an ID, and the ID of the current partition can be an integer greater than or equal to zero. There are a total of T current partitions, and the IDs of the T current partitions form an equidistant sequence in which the first term is 0, the tolerance is 1, and the number of terms is T. Each deterministic partition has an ID, and the deterministic partition ID may be an integer greater than or equal to zero. There are a total of S definite partitions, and the IDs of the S definite partitions form an equidistant sequence in which the first term is 0, the difference is 1, and the number of terms is S. For example, divide 12 partitions into 24 partitions, where the partition IDs before split are 0, 1, 2, 3, ..., 9, 10, and 11, and after split The partition IDs are 0, 1, 2, 3, ..., 21, 22, and 23.

  After partitioning, the partition ID generation rule is that the partition acquired after partitioning has one partition holding the original ID of the partition, the ID value of the other partition, the original ID value of the partition, etc. It may be to form a difference sequence, where the terms in the sequence increase gradually with a tolerance of the total number M of partitions before splitting. For example, there are a total of 200 partitions before splitting, and after splitting, each partition is split into 3 partitions, and the IDs of the 3 partitions generated after splitting the partition with ID 21 are , 221, 421, and 621 in this order. If the ID of the current partition is 0 in the first term, the tolerance is 1, and the incremental number sequence is further increased after the whole split process is completed, this ID generation rule may be changed. .

  For example, according to another partition ID generation rule, IDs of three partitions obtained by dividing a partition with ID 0 can be further 0, 201, and 202, and ID 1 is The partition IDs obtained by splitting the partitions are 1, 203, and 204, the partition IDs obtained by splitting the partition with ID 3 are 3, 205, and 206, and others The partition ID can be obtained in the same way.

  If the total number S of current partitions is about to reach the total number L of defined partitions, the following case can occur: the total number of partitions generated by dividing each partition once is the fixed partition If the total number of partitions is greater than L, the total number of partitions can be increased by splitting, so that only a part of the partition is split or the split factor is reduced so that the total number of partitions does not exceed the total number L of committed partitions. Note that it can be.

  Step 25: Part of the partitions owned by the original M IP disks are newly added to the N IPs so that the M / P partitions are evenly distributed on the M + N IP disks Perform partition migration to migrate. After step 25 is executed, the total number of partitions in the entire system does not change, so the average number of partitions owned by each IP disk is reduced. Step 24 and step 25 are executed alternatively. In all embodiments of the partition management method, step 25 or step 26 is not performed after step 24 is performed.

  Step 26: Update the total number of current IP disks recorded in the controller to M + N and update the total number of current partitions to S. This step may be executed simultaneously with step 24. The number of current partitions on each IP disk is approximately S / (M + N), so it is not necessary to record the total number of partitions S; instead, the number of current partitions on each IP disk in the cluster is Record approximately S / (M + N).

  Step 26 is preparation for the next division, and therefore step 26 is not an essential step for the current partition management process.

  Note that S current partitions are allocated to M + N IP disks. The process of updating the partition view may be further executed. Here, the partition view records the IP disk corresponding to the current partition, and particularly the association between the ID of the current partition and the IP disk address. It can be. The process of updating the partition view may be executed in step 24 or step 26. The following data processing method may use the partition view updated in this step. In fact, in another embodiment of the present invention, when a change to the current partition is caused, the partition can only be used after the association between the current partition and the IP disk has been recorded correctly. In this case, a process for updating the partition view needs to be executed.

  The partition management method described above may be executed using the hardware shown in FIG. In FIG. 3, the controller 3 includes an interface 31, a processor 32, and a storage medium 33.

  The interface 31 is configured to provide an external interface to connect to, for example, a storage disk cluster and a management server. The storage medium 33 is configured to store computer program code. The processor 32 executes the partition management method described above by operating the program code in the storage medium 33.

  Referring to FIG. 4, the embodiment of the present invention further provides a partition management device 4, wherein the partition management device 4 may be hardware or virtual hardware formed by software. . The partition management device 4 can execute the partition management method described above. The partition management device 4 includes a storage disk detection module 41, a first preset condition determination module 42, and a partition division module 43. The partition management device 4 may further include an initialization module 40 as necessary. The partition management device 4 may further include an update module 44 as necessary.

  The initialization module 40 determines the number L of fixed partitions that are natural numbers according to the maximum number of IP disks in the cluster, and determines the number of initial partitions owned by each IP disk according to the number of initial IP disks. Composed. This step is a pre-step that is performed only when the controller is initialized, and is therefore optional.

  The update module 44 may record the initial IP disk quantity as the current IP disk quantity M and record the initial partition quantity owned by each IP disk as the current partition quantity P, where The total number T of existing partitions is T = M · P. The update module 44 further records the quantity L of confirmed partitions. Thereafter, when the number of IP disks in the cluster changes or the number of partitions of each IP disk changes, the number M of current IP disks and the number P of current partitions are updated. In the embodiment of the present invention, the symbol “•” represents a product.

  The number of deterministic partitions is constant, and deterministic partitions cannot be divided and can be perceived by the user. An initial partition is generally a parent partition, and a parent partition is a partition that can be split to generate a next stage parent partition or a deterministic partition by splitting. The parent partition is used by the storage system and the user cannot perceive the presence of the parent partition.

  The value of L can be set by the user or can be automatically allocated by the system and is generally dependent on both the maximum number of IP disks and the quantity of each IP disk when the number of IP disks is the maximum That is, L = the maximum number of IP disks × the number of partitions of each IP disk. Assume that owning X partitions is the desired value for each partition. For example, if each IP disk owns 100 partitions, that is, the user thinks that X = 100 is a relatively appropriate value, the maximum number of IP disks that an IP disk cluster can support is As a result, L = 10,000 × 100 = 1,000,000.

  Similarly, the initial partition quantity can be set by the user or automatically assigned by the system. One optional scheme is that in the initial situation, assuming that the number M of IP disks is M = 75 and 100 partitions are allocated to each partition, the result is the total number of initial partitions = 75 × 100 = 7,500. By using the method of the present invention, in the initial stage, the number of partitions owned by each IP disk in the storage system may be freely set, where the number of partitions meets the user's requirements. A partition may be set to a value that does not occupy more resources than necessary, such as storage resources, computational resources, and bandwidth resources.

  When the storage disk detection module 41 detects that N new storage disks are about to be added to the cluster, it obtains the number M of current storage disks in the cluster and the total number T of partitions currently in the cluster. Where M, N, and T are all natural numbers.

  The partition management device 4 is connected to the IP disk cluster. Therefore, the storage disk detection module 41 can detect the IP disk newly added to the cluster. At present, there are already M IP disks, and the total number of current partitions is MP. The N IP disks are physically connected to the cluster and can be detected by the storage disk detection module 41, but data cannot be stored because the partition is not allocated to the IP disk.

  Note that “current” refers to the point in time when the process is about to be executed. In the present embodiment, no IP disk is added after the partition management device 4 is initialized. Therefore, the number of current IP disks is M. In another embodiment, if the number of IP disks in the IP disk cluster has changed before this step is performed, the number of current IP disks is not M. If the partition was partitioned before this step was executed, the number of initial partitions currently owned by each IP disk is greater than P. T partitions are allocated approximately evenly across M IP disks.

  The first preset condition determination module 42 is configured to determine whether the mathematical relationship between the total number T of current partitions and the total number M + N of storage disks satisfies the first preset condition.

  Determine whether the current partition quantity meets the common requirements of the current IP disk and the newly added IP disk. That is, determine whether the mathematical relationship between the total number T of current partitions and the total number M + N of storage disks satisfies the first preset condition, where the determination is the formula T / (M + N ) And the first threshold. The division threshold value may be preset by the controller. When this process is executed for the first time after the controller is initialized, T = M · P. The first threshold may be an integer greater than 10 and less than 20, such as 16, 17, 18, 19, or 20.

  In one determination method, when T / (M + N) is less than the first threshold, the first preset condition is satisfied, and division is necessary. If the number of partitions owned by each IP disk is less than the partitioning threshold (or can be set below the partitioning threshold), the number of partitions owned by each IP disk will be too small without performing partitioning. As a result, at least one current partition needs to be split. If the number of partitions owned by each IP disk is not less than the partitioning threshold, partition migration must be performed.

  Another determination method is that if the average number of partitions owned by each IP disk is greater than the threshold (or can be set to be higher than the threshold) after the first partitioning, each IP disk can be divided. Indicates that the number of partitions owned by is excessively large, and as a result, partition migration must be performed. If the average number of partitions owned by each IP disk is not greater than the threshold, at least one current partition must be split. In addition, the two decision methods may also be combined to make a decision, and the solution with the highest satisfaction with the service is selected. All of the following embodiments illustrate the present invention by using the first determination method as an example.

  For the partitioning module 43, if the result of the determination by the first preset condition determining module 42 is that the relationship satisfies the first preset condition, the partitioning module 43 acquires the partition acquired after the partitioning. It is configured to divide at least one current partition so that the total number of S becomes S, and allocate the partitions acquired after the division to M + N storage disks. The mathematical relationship between the total number of partitions S acquired after splitting and the total number of storage disks M + N satisfies the second preset condition, and the total number of partitions acquired after splitting is the total number of confirmed partitions supported by the cluster Not greater than L, both L and S are natural numbers greater than 1. The partitions obtained after the division can be evenly allocated to M + N storage disks, or are approximately equally allocated to M + N storage disks.

  At least one current partition can be split once or can be split multiple times until the quantity of partitions obtained after the split meets the requirement. The mathematical relationship between the total number of partitions obtained after splitting S and the total number of storage disks M + N satisfies the second preset condition, in particular, S / (M + N) is greater than or equal to the second threshold And the second threshold may be a natural number. The second threshold may be a natural number greater than 25 and less than 50, such as 47, 48, 49, or 50, for example.

  There can be various schemes for determining whether the quantity of partitions obtained after splitting meets the requirement. For example, when the mathematical relationship between the total number S of partitions acquired after the division and the total number M + N of the storage disks satisfies the second preset condition, the division ends. Specifically, for example, if the average number of partitions owned by each divided IP disk is greater than or equal to the second threshold value, it is considered that the second preset condition is satisfied, and the continuation of the division ends. To do. Alternatively, if the average number of partitions owned by each divided IP disk satisfies the preset threshold range, it is considered that the requirement is satisfied.

When each partition is divided into a plurality of subpartitions in each division, each partition is divided into the same number of subpartitions. The magnification of the change in the partition from the first division to the end of the division is called a division coefficient. Assume that each parent partition is split into two subpartitions in each split. As a result, when the division after the first division is completed, the decimation factor is 2, when the split after the second division is completed, the decimation factor is 2 ^2.

  If dividing at least one current partition indicates dividing all current partitions, the total number of partitions obtained after the division is S = T × the division coefficient. When dividing at least one current partition indicates dividing some current partitions, the total number of partitions obtained after the division is S <T × division coefficient. The former division method has a higher division speed, and the latter division method has a better adjustment function of the total number of partitions obtained after the division. For specific details, refer to the description of the embodiment of the partition management method. For convenience of explanation, this embodiment of the present invention will be described using the former division method.

  An example of the latter division method is shown below. Assuming that the total number of deterministic partitions is 1,000, the total number of current partitions is 512, and the division factor is 2, dividing each partition will result in 1,024 partitions exceeding the total number of deterministic partitions after being divided Will be acquired. The total number of current partitions is not allowed to be greater than the total number of committed partitions. To avoid such a case, only 488 current partitions can be split, i.e. 488 + 512 = 1,000, i.e. the total number of current partitions obtained after splitting has just reached 1,000, and It will not be larger than the total value. In addition, if the user thinks that 1,000 current partitions are an excessively large quantity, for example, if the user thinks that it is most appropriate that the total number of current partitions obtained after splitting reaches 800, then splitting The 288 partitions of the 512 current partitions can be divided according to the division factor 2 so that the total number of current partitions acquired later just reaches 800. Using the latter partitioning method, it may be that only one partition is split each time, and when the partitioning factor is 2, the difference in the total number of current partitions before and after the partitioning is 1, so The granularity for partitioning is minimal.

  In addition, the division coefficient may be changed based on the above-described embodiment. For example, 2 is used as the division coefficient during the first division, but the division coefficient is changed to 5 during the second division. This can also achieve the purpose of making the value of the total number of partitions obtained after the division more adjustable. There is an association between the current partition and the determined partition, and this association can be stored by the update module 44 or further communicated to the data processing device by the update module 44. Each current partition has an ID, and the ID of the current partition can be an integer greater than or equal to zero. The IDs of all current partitions have an initial number of 0, a tolerance of 1, and form an arithmetic sequence. Each deterministic partition has an ID, and the deterministic partition ID may be an integer greater than or equal to zero. All deterministic partition IDs have an initial difference of 0 and a difference of 1 to form an equidistant sequence. For example, divide 12 partitions into 24 partitions, where the partition IDs before split are 0, 1, 2, 3, ..., 9, 10, and 11, and after split The partition IDs are 0, 1, 2, 3, ..., 21, 22, and 23.

  After partitioning, the partition ID generation rule is that the partition acquired after partitioning has one partition holding the original ID of the partition, the ID value of the other partition, the original ID value of the partition, etc. It may be to form a difference number sequence, wherein the terms of the difference number sequence increase gradually with a tolerance of the total number M of partitions before the division. For example, there are a total of 200 partitions before splitting, and after splitting, each partition is split into 3 partitions, and the IDs of the 3 partitions generated after splitting the partition with ID 21 are , 221, 421, and 621 in this order. If the ID of the current partition is 0 in the first term, the tolerance is 1, and the incremental number sequence is further increased after the whole split process is completed, this ID generation rule may be changed. .

  If the total number S of current partitions is about to reach the total number L of defined partitions, the following case can occur: the total number of partitions generated by dividing each partition once is the fixed partition If the total number of partitions is greater than L, the total number of partitions can be increased by splitting, so that only a part of the partition is split or the split factor is reduced so that the total number of partitions does not exceed the total number L of committed partitions. Note that it can be.

  The update module 44 updates the total number of current IP disks recorded in the partition management device 4 to M + N and updates the total number of current partitions to S after the processing of the partitioning module 43 is executed. Composed. The number of current partitions on each IP disk is approximately S / (M + N), so it is not necessary to record the total number of partitions S; instead, the number of current partitions on each IP disk in the cluster is Record approximately S / (M + N).

  The processing performed by the update module 44 is preparation for the next division, and therefore the update module 44 is not an essential module for this partition management processing.

  If necessary, the partitioning module 43 or the update module 44 may further execute processing related to the partition view, where the partition view indicates that the IP disk corresponds to the current partition, for example, the current partition. Record the correspondence between the partition ID of the disk and the corresponding IP disk address of the IP disk. That is, the partition view records which of the M + N IP disks corresponds to each partition of the S current partitions. The following data processing apparatus can use the updated partition view.

  A partition migration module (not shown) may further be included as necessary. If partitioning has not been performed, the partition migration module will ensure that the partitions owned by the M original IP disks are distributed so that the M / P partitions are evenly distributed over the M + N IP disks. Can be used to perform partition migration to migrate some to the newly added N IPs.

  Referring to FIG. 5, the present invention further provides a data processing method applied to a partition management device. The partition management device stores the partition view, and the partition view records the association between the ID of the current partition and the address of the storage disk (eg, IP disk). The data processing method is executed after the partition management method and is based on the partition management method. However, the two methods are relatively independent. The partition management device is connected to the controller. The partition management device is, for example, a management server or a switch cluster. This will be described below using the management server as an example. The embodiment of the data processing method may be executed based on the partition view provided in the partition management method described above. The partition view is generated by the controller and sent to the partition management device in the partition management device cluster for storage.

  Step 51: Generate key-value data according to data to be written. For example, data to be written is divided to obtain a set including value, and a key of value is generated to form key value data, where key value is a combination of a key and a value corresponding to the key It is. Since a part of data to be written can be divided into a plurality of values, a plurality of key values can be generated correspondingly. For the sake of brevity, the specific key-value process is described with only the following steps.

  The data to be written, for example a file or data stream, is from the application server. When the size (Size) of data to be written is relatively large, the management server can divide the data in order to facilitate storage. For example, the management server may divide the data into equally sized data segments of 1 MB each, and each segment is referred to as a value. The key is used to uniquely identify the value. Therefore, different value keys are different. For example, “data file name + numerical number” may be used as a value key. It is not always necessary to divide data having a relatively small size, and a key of data is directly generated, and then key-value data is formed. In some special scenarios, it is not necessary to split large size data, the corresponding key-value data is formed directly and then sent to the IP disk for storage.

  Step 52: The key in the key value data is acquired, and the ID of the confirmed partition corresponding to the key value data is calculated according to the key. As described above, the key value data includes a value and a key that uniquely corresponds to the value.

  The method for calculating the ID of the deterministic partition is to perform a hash operation on the key to obtain the hash value of the key, and perform a modulo operation on the hash value according to the total number L of the deterministic partitions, The remainder is used as the ID of the deterministic partition, and L is a natural number of 2 or more. Thus, the acquired ID of the confirmed partition is a digital number. In another embodiment, the equivalent transformation is to map the digital number to another code, for example, an English number, using the English number as the ID of the deterministic partition. In addition to the deterministic partition, the initial and current partitions may also be represented by using numbers in English notation. When calculating the deterministic partition corresponding to the key, or when calculating the association between the current partition and the deterministic partition, the English number may be remapped to a digital number. The digital number obtained by mapping is counted like the partition ID of the numerical format using the “method”. For the concept of the deterministic partition in this algorithm, refer to the description in the above embodiment of the partition management method.

  Step 53: Calculate the ID of the current partition corresponding to the ID of the confirmed partition. Here, the ID of each current partition corresponds to the ID of a plurality of confirmed partitions.

  The algorithm for calculating the current partition of the deterministic partition is to perform a modulo operation on the ID of the deterministic partition according to the total number T of the current partition, the remainder is used as the ID of the current partition, and the total number T of the current partition is , A natural number. Similarly, the ID of the current partition is a digital number, and the equivalent conversion is to map the digital number to another code as the ID of the current partition. The current partition refers to a partition currently owned by the IP disk cluster, and each current partition corresponds to one IP disk. Reference may be made to the description of the current partition in the previous embodiment of the partition management method. The fixed partition is a subpartition of the current partition.

  In view of such circumstances, there is a correspondence between the ID of the current partition and the IDs of a plurality of confirmed partitions. For details on the association, refer to the embodiment of the partition management method. An association exists between the current partition and the committed partition, and this association can be stored in the controller, read when executing step 53, or the association is stored in advance. It is not necessary, and when step 53 is executed, it is obtained by calculation according to an algorithm. Each current partition has an ID, and the ID of the current partition can be an integer greater than or equal to zero. The set of IDs for all current partitions has an initial term of 0 and a tolerance of 1 to form an arithmetic sequence. Each deterministic partition has an ID, and the deterministic partition ID may be an integer greater than or equal to zero. The set of all deterministic partition IDs is 0 for the first term and 1 for the difference, forming an arithmetic sequence. The method for obtaining a deterministic partition corresponding to the current partition is to perform a modulo operation on the total number of current partitions using the deterministic partition ID, and the remainder value is the current value corresponding to the deterministic partition. Used as partition ID.

  In addition, after splitting a partition, the ID generation rule keeps the original ID of the partition in the partition acquired after splitting, the ID value of the other partition and the original ID value of the partition. May form an arithmetic sequence, where the terms of the sequence increase incrementally with the tolerance of the total number of partitions before partitioning. For example, there are a total of 200 partitions before splitting, and after splitting, each partition is split into 3 partitions, and the IDs of the 3 partitions generated after splitting the partition with ID 21 are , 221, 421, and 621 in this order. If the ID of the current partition is 0 in the first term, the tolerance is 1, and the incremental number sequence is further increased after the whole split process is completed, this ID generation rule may be changed. . For example, the partition whose ID is 0 is divided according to another division method. The IDs of the three partitions obtained after the split can be further 0, 201, and 202, and the IDs of the partitions obtained by splitting the partition whose ID is 1 are 1, 203, and 204 The IDs of the partitions obtained by dividing the partition whose ID is 3 are 3, 205, and 206, and the IDs of the other partitions can be obtained in the same manner.

  Step 54: Search the partition view to get the address of the storage disk corresponding to the current partition ID.

  The partition management device stores the partition view, and the partition view records the association between the ID of the current partition and the address of the storage disk. When the storage disk is an IP disk, the address of the storage disk can be an IP address. If the storage disk is based on another type of protocol, for example ATM or IPX protocol, the address of the storage disk is an ATM address or an IPX address.

  Step 55: Generate a key-value packet by using the address of the storage disk as the destination address and send the key-value packet to the storage disk, where the payload of the key-value packet carries the key-value data.

  After receiving the key value packet, the storage disk stores the key value data.

As shown in FIG. 6, FIG. 6 is an embodiment of a data processing device according to the present invention. The data processing device is a memory 61 configured to store a partition view, the partition view records an association between the ID of the current partition and the address of the storage disk, the memory 61, and an external An interface 62 configured to provide an interface; a computer readable medium 63 configured to store a computer program; and a program connected to the memory 61, the interface 62, and the computer readable medium 63 And a processor 64 configured to perform the above-described data processing method.
The data processing method is, for example, a step of acquiring a key in key-value data and calculating the ID of a confirmed partition corresponding to the key-value data according to the key. A step including a corresponding key and a step of calculating an ID of a current partition corresponding to an ID of a confirmed partition, wherein each current partition ID corresponds to an ID of a plurality of confirmed partitions; In order to obtain the address of the storage disk corresponding to the ID of the current partition, a key value packet is generated by searching the partition view, and using the address of the storage disk as the destination address. The stress And transmitting to the di-disk, the key-value packet, carrying the key value data, including a step.

  As shown in FIG. 7, FIG. 7 is an accompanying drawing of a data processing device 7 according to an embodiment of the present invention. The data processing device 7 includes a storage module 71, a confirmed partition calculation module 6, a current partition calculation module 73, a search module 74, and a transmission module 75. The data processing device 7 may further include a key / value data generation module 76 as necessary.

  The storage module 71 is configured to store the partition view, and the partition view records the association between the ID of the current partition and the address of the storage disk. The storage medium used by the storage module 71 can be a flash memory or a hard disk.

  The partition view of the storage module 71 is from a partition management device, for example, the partition management device 4 in FIG. The storage module 71 can be connected to the partition management device 4 to receive the partition view.

  The confirmed partition calculation module 72 is configured to obtain a key in the key-value data and calculate an ID of the confirmed partition corresponding to the key-value data according to the key, where the key-value data includes the value and the value Contains a key that uniquely corresponds to.

  The method used by the deterministic partition calculation module 72 to calculate the ID of the deterministic partition is to perform a hash operation on the key to obtain the hash value of the key and hash the modulo operation according to the total number L of deterministic partitions. The remainder is used as the ID of the deterministic partition, and L is a natural number of 2 or more. Thus, the acquired ID of the confirmed partition is a digital number. In another embodiment, the equivalent transformation is to map the digital number to another code, for example, an English number, using the English number as the ID of the deterministic partition. In addition to the deterministic partition, the initial and current partitions may also be represented by using numbers in English notation. When calculating the deterministic partition corresponding to the key, or when calculating the association between the current partition and the deterministic partition, the English number may be remapped to a digital number. The digital number obtained by mapping is counted like the partition ID of the numerical format using the “method”. For the concept of the definite partition, refer to the description in the above embodiment of the partition management method.

  The current partition calculation module 73 is configured to calculate the ID of the current partition corresponding to the ID of the confirmed partition, and the ID of each current partition corresponds to the IDs of a plurality of confirmed partitions.

  The method used by the current partition calculation module 73 to calculate the ID of the current partition corresponding to the ID of the deterministic partition is to perform a modulo operation on the ID of the deterministic partition according to the total number T of current partitions, The remainder is used as the ID of the current partition, and the total number T of current partitions is a natural number. Similarly, the ID of the current partition is a digital number, and the equivalent conversion is to map the digital number to another code as the ID of the current partition. The current partition indicates a partition owned by the current IP disk cluster, and each current partition corresponds to one IP disk. Reference may be made to the description of the current partition in the previous embodiment of the partition management method. The fixed partition is a subpartition of the current partition.

  In view of such circumstances, there is a correspondence between the ID of the current partition and the IDs of a plurality of confirmed partitions. For details regarding the association, refer to the embodiment of the partition management apparatus. After generating the association, the partition management device informs each data processing device of the association. An association exists between the current partition and the final partition, and this association can be stored in the current partition calculation module 73, or the association does not have to be stored in advance, It may be obtained by calculation by the partition calculation module 73. Each current partition has an ID, and the ID of the current partition can be an integer greater than or equal to zero. The set of IDs of all current partitions can form an increasing equidistant sequence with an initial term of 0 and a tolerance of 1. Each deterministic partition has an ID, and the deterministic partition ID may be an integer greater than or equal to zero. The set of IDs for all deterministic partitions forms an incremental sequence with 0 as the first term and 1 as the difference. For example, divide 12 partitions into 24 partitions, where the partition IDs before split are 0, 1, 2, 3, ..., 9, 10, and 11, and after split The partition IDs are 0, 1, 2, 3, ..., 21, 22, and 23.

  In addition, after splitting a partition, the ID generation rule keeps the original ID of the partition in the partition acquired after splitting, the ID value of the other partition and the original ID value of the partition. May form an arithmetic sequence, where the terms of the sequence increase incrementally with the tolerance of the total number of partitions before partitioning. For example, there are a total of 200 partitions before splitting, and after splitting, each partition is split into 3 partitions, and the IDs of the 3 partitions generated after splitting the partition with ID 21 are , 221, 421, and 621 in this order. If the ID of the current partition is 0 in the first term, the tolerance is 1, and the incremental number sequence is further increased after the whole split process is completed, this ID generation rule may be changed. . For example, the partition whose ID is 0 is divided according to another division method. The IDs of the three partitions obtained after the split can be further 0, 201, and 202, and the IDs of the partitions obtained by splitting the partition whose ID is 1 are 1, 203, and 204 The IDs of the partitions obtained by dividing the partition whose ID is 3 are 3, 205, and 206, and the IDs of the other partitions can be obtained in the same manner.

  The search module 74 is configured to search the partition view 71 stored by the storage module to obtain the address of the storage disk corresponding to the current partition ID.

  The partition view records the association between the ID of the current partition and the address of the storage disk. When the storage disk is an IP disk, the address of the storage disk can be an IP address. If the storage disk is based on another type of protocol, for example, an ATM or IPX protocol, the storage disk address is an ATM address or an IPX address.

  A sending module 75 configured to generate a key-value packet by using the address of the storage disk as a destination address and send the key-value packet to the storage disk, where the key-value packet is key-value data Transport.

  Next, as a destination device for the key value packet, the storage disk is configured to use the switch cluster to receive the key value packet and then store the key value data.

  The key-value data generation module 76 is configured to generate key-value data, for example, to divide the data to be written to obtain a set including values, and to create a key-value data Here, the key value is a combination of a key and a value corresponding to the key. Since a part of data to be written can be divided into a plurality of values, a plurality of key values can be generated correspondingly. For the sake of brevity, this embodiment of the present invention describes only a specific key-value processing scheme.

  The data to be written, for example a file or data stream, is from the application server. When the size (Size) of data to be written is relatively large, the management server can divide the data in order to facilitate storage. For example, the management server may divide the data into equally sized data segments of 1 MB each, and each segment is referred to as a value. The key is used to uniquely identify the value. Therefore, different value keys are different. For example, “data file name + numerical number” may be used as a value key. It is not necessary to divide data of a relatively small size, and the data is directly used as value in order to generate key-value data. In some special scenarios, large size data does not necessarily need to be split and is used directly as value to generate key-value data.

  When the data processing device 7 includes the key value data generation module 76, the confirmed partition calculation module 72 can be connected to the key value data generation module 76, and the data processing device 7 includes the key value data generation module 76. If not, the confirmed partition calculation module 72 may obtain the key value data directly from the application server via the external interface.

  Those skilled in the art will appreciate that each aspect of the invention or possible embodiments of each aspect may be specifically implemented as a system, method, or computer program product. Accordingly, each aspect or possible implementation of each aspect of the present invention is described in terms of hardware-only embodiments, software-only implementations, which are uniformly referred to herein as "circuits", "modules", or "systems". Forms (including firmware, resident software, etc.) or form of embodiment using a combination of software and hardware may be used. In addition, each aspect of the invention or possible implementations of each aspect may take the form of a computer program product, wherein the computer program product is stored on a computer readable medium. Point to.

  The computer readable medium may be a computer readable signal medium or a computer readable storage medium. Computer readable storage media include random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, and compact disk read only memory (CD-ROM), This includes but is not limited to electronic, magnetic, optical, electromagnetic, infrared, and semiconductor systems, devices, or apparatus, or any suitable combination thereof.

  A processor in the computer reads the computer readable program code stored on the computer readable medium so that the processor can perform the functions and operations defined by each step or combination of steps in the flowchart. , Generated to perform the functions and operations specified in each block or combination of blocks in the block diagram.

11 Application server
12 Management server cluster
13 Switch cluster
14 IP disk cluster
15 Controller
121 Management server
141 IP disk
3 Controller
31 interface
32 processor
33 Storage media
4 Partition management unit
40 Initialization module
41 Storage disk detection module
42 First preset condition determination module
43 Partitioning module
44 Update module
6 Data processor
61 memory
62 Interface
63 Computer-readable media
64 processors
7 Data processing equipment
71 Memory module
72 Deterministic partition calculation module
73 Current partition calculation module
74 Search module
75 Transmitter module
76 Key-value data generation module

The application server 11 transmits a data read command or a data write command. The management server 121 is, for example, a distributed object pool (distributed object pool, DOP), which provides an interface object to the application server 11, wherein the object may be very large, for example, the basic unit May be GB.

If the object is too large, the management server 121 may divide the object into small segments. For example, the management server 121 may divide the object into segments each having a size of 1 MB, where each segment is a value and each value has a tag called a key. The management server 121 can perform a hash operation on the key and perform matching between the calculated hash value and the partition. In addition, the management server 121 may further store a partition view, where the partition view may be an association between the ID of the current partition and the IP disk, in particular the correspondence between the IP disk and the partition. Record the date. Therefore, the management server 121 can search for an IP disk corresponding to each key value. The IP disk to be found is referred to as a target disk, and the management server 121 can generate an IP packet by using the address of the target IP disk as a destination address and transmit the IP packet to the switch cluster 13. For example, the contents recorded in the partition view are shown in Table 1, where i is a natural number and m is a natural number greater than 1.

The switch cluster 13 includes at least one switch, and is used for exchanging data between the management server 121 and the IP disk 141 during data reading or data writing. The process of retrieving the partition view is also left to the switches in the switch cluster 13 and may be performed by the switches in the switch cluster 13, that is, the switches in the switch cluster 13 store the partition view. In this scenario, when data is transmitted to the switch cluster 13 , the management server 121 may use a new type of packet that can be referred to as a key-value packet without using an IP packet. The difference between the key value packet and the IP packet is that the destination address of the key value packet is the partition number obtained by calculation using the key, and the packet type field distinguishes between the IP packet and the key value packet. The other part of the key-value packet is the same as that of the IP packet. The switch searches the mapping table between the partition number and the IP disk address, where the mapping table is stored by the switch, replaces the destination address with the IP disk address corresponding to the partition, and the key-value packet The packet type is modified to convert the IP packet into an IP packet, and then transferred to the corresponding IP disk for storing the IP packet.

When the application server 11 reads data or needs to write data, the data sequentially reaches the IP disk through the management server 121 and the switch cluster 13 .

If dividing at least one current partition indicates dividing all current partitions, the total number of partitions obtained after the division is S = T × the division coefficient. When dividing at least one current partition indicates dividing some current partitions, the total number of partitions obtained after the division is S <T × division coefficient. The former division method has a higher division speed, and the latter division method has a better adjustment function of the total number of partitions obtained after the division. For convenience of explanation, this embodiment of the present invention will be described using the latter division method.

If the total number T of current partitions is about to reach the total number L of defined partitions, the following case can occur: the total number of partitions generated by dividing each partition once is the fixed partition If the total number of partitions is greater than L, the total number of partitions can be increased by splitting, so that only a part of the partition is split or the split factor is reduced so that the total number of partitions does not exceed the total number L of committed partitions. Note that it can be.

Step 25: N IP disks newly added with some of the partitions owned by the original M IP disks so that the M / P partitions are evenly distributed on the M + N IP disks Perform partition migration to migrate to. After step 25 is executed, the total number of partitions in the entire system does not change, so the average number of partitions owned by each IP disk is reduced. Step 24 and step 25 are executed alternatively. In all embodiments of the partition management method, step 25 or step 26 is not performed after step 24 is performed.

The value of L can be set by the user or can be automatically allocated by the system and is generally dependent on both the maximum number of IP disks and the quantity of each IP disk when the number of IP disks is the maximum That is, L = the maximum number of IP disks × the number of partitions of each IP disk. Assume that owning X partitions is the desired value for each IP disk . For example, if each IP disk owns 100 partitions, that is, the user thinks that X = 100 is a relatively appropriate value, the maximum number of IP disks that an IP disk cluster can support is As a result, L = 10,000 × 100 = 1,000,000.

Similarly, the initial partition quantity can be set by the user or automatically assigned by the system. One optional scheme is that in the initial situation, assuming that the number of IP disks M is M = 75 and 100 partitions are allocated to each IP disk , the result is that the total number of initial partitions = 75 × 100 = 7,500. By using the method of the present invention, in the initial stage, the number of partitions owned by each IP disk in the storage system may be freely set, where the number of partitions meets the user's requirements. A partition may be set to a value that does not occupy more resources than necessary, such as storage resources, computational resources, and bandwidth resources.

If dividing at least one current partition indicates dividing all current partitions, the total number of partitions obtained after the division is S = T × the division coefficient. When dividing at least one current partition indicates dividing some current partitions, the total number of partitions obtained after the division is S <T × division coefficient. The former division method has a higher division speed, and the latter division method has a better adjustment function of the total number of partitions obtained after the division. For specific details, refer to the description of the embodiment of the partition management method. For convenience of explanation, this embodiment of the present invention will be described using the latter division method.

If the total number T of current partitions is about to reach the total number L of defined partitions, the following case can occur: the total number of partitions generated by dividing each partition once is the fixed partition If the total number of partitions is greater than L, the total number of partitions can be increased by splitting, so that only a part of the partition is split or the split factor is reduced so that the total number of partitions does not exceed the total number L of committed partitions. Note that it can be.

If necessary, the partitioning module 43 or the update module 44 may further execute a process of updating the partition view, where the partition view indicates that the IP disk corresponds to the current partition, for example, Records the correspondence between the partition ID of the current partition and the corresponding IP disk address of the IP disk. That is, the partition view records which of the M + N IP disks corresponds to each partition of the S current partitions. The following data processing apparatus can use the updated partition view.

A partition migration module (not shown) may further be included as necessary. If partitioning has not been performed, the partition migration module will ensure that the partitions owned by the M original IP disks are distributed so that the M / P partitions are evenly distributed over the M + N IP disks. Can be used to perform partition migration to migrate part to N newly added IP disks .

Algorithm for calculating the current partition ID of the defined partition ID, in accordance with the total number of the current partition T, and by performing a modulo operation on the ID of the determined partition remainder is used as the ID of the current partition, the total number of the current partition T is a natural number. Similarly, the ID of the current partition is a digital number, and the equivalent conversion is to map the digital number to another code as the ID of the current partition. The current partition refers to a partition currently owned by the IP disk cluster, and each current partition corresponds to one IP disk. Reference may be made to the description of the current partition in the previous embodiment of the partition management method. The fixed partition is a subpartition of the current partition.

As shown in FIG. 7, FIG. 7 is an accompanying drawing of a data processing device 7 according to an embodiment of the present invention. The data processing device 7 includes a storage module 71, a confirmed partition calculation module 72 , a current partition calculation module 73, a search module 74, and a transmission module 75. The data processing device 7 may further include a key / value data generation module 76 as necessary.

Claims (32)

A data processing method applied to a partition management device, wherein the partition management device stores a partition view, and the partition view records a correspondence between an ID of a current partition and an address of a storage disk, The method
Obtaining a key in the key-value data and calculating an ID of a confirmed partition corresponding to the key-value data according to the key, wherein the key-value data uniquely corresponds to the value and the value Including the key being, and
Calculating an ID of a current partition corresponding to the ID of the defined partition, wherein each current partition ID corresponds to an ID of a plurality of confirmed partitions; and
Searching the partition view to obtain the address of the storage disk corresponding to the ID of the current partition;
Generating a key-value packet by using the address of the storage disk as a destination address, and transmitting the key-value packet to the storage disk, wherein the key-value packet carries the key-value data A data processing method comprising the steps of: The partition management device further records the total number L of confirmed partitions, and calculates the confirmed partition ID corresponding to the key-value data according to the key,
Performing a hash operation on the key to obtain a hash value of the key, and performing a modulo operation on the hash value according to the total number L of deterministic partitions, wherein a remainder is a value of the deterministic partition; The data processing method according to claim 1, further comprising a step used as said ID. The partition management device further records the total number T of current partitions, and calculates the ID of the current partition corresponding to the ID of the confirmed partition,
Performing a modulo operation on the ID of the defined partition according to the total number T of current partitions, wherein a remainder is used as the ID of the current partition, and the ID of the defined partition is greater than or equal to 0 3. The data processing method according to claim 1 or 2, particularly including a step that is an integer.   2. The data processing method according to claim 1, wherein the determined partition is a sub-partition of the current partition and is acquired by dividing the current partition. The data processing method is in particular a data writing method, and prior to the step of obtaining a key in the key value data, the method comprises:
The data processing according to claim 1, further comprising: dividing the data to be written to obtain a value set including the value, and generating the key of the value to form the key value data. Method. A data processing device, the device comprising:
A storage module configured to store a partition view, wherein the partition view records an association between an ID of a current partition and an address of a storage disk;
A confirmed partition calculation module configured to obtain a key in key-value data and calculate an ID of a confirmed partition corresponding to the key-value data according to the key, wherein the key-value data is a value And a deterministic partition calculation module, including the key uniquely corresponding to the value;
A current partition calculation module configured to calculate an ID of a current partition corresponding to the ID of the defined partition, wherein each current partition ID corresponds to a plurality of confirmed partition IDs, Partition calculation module;
A search module configured to search the partition view stored by the storage module to obtain an address of a storage disk corresponding to the ID of the current partition;
A transmission module configured to generate a key-value packet by using the address of the storage disk as a destination address and to transmit the key-value packet to the storage disk, wherein the key-value packet is A data processing apparatus comprising: a transmission module that carries the key-value data. The storage module is further configured to record a total number L of defined partitions, and the defined partition calculation module is
It is particularly configured to perform a hash operation on the key to obtain a hash value of the key, and to perform a modulo operation on the hash value according to the total number L of deterministic partitions, wherein a remainder is the deterministic partition 7. The data processing device according to claim 6, wherein the data processing device is used as the ID. The storage module is further configured to record a total number T of current partitions, and the current partition calculation module includes:
Specifically configured to perform a modulo operation on the ID of the deterministic partition according to the total number T of current partitions, a remainder is used as the ID of the current partition, and the ID of the deterministic partition is greater than or equal to zero The data processing apparatus according to claim 6 or 7, wherein the data processing apparatus is an integer.   7. The data processing apparatus according to claim 6, wherein the determined partition is a sub-partition of the current partition and is acquired by dividing the current partition. The data processing device includes:
A key value data generation module configured to divide data to be written to obtain a value set including the value and generate the key of the value to form the key value data; The data processing apparatus according to claim 6, further comprising: A data processing device, the data processing device comprising:
A memory configured to store a partition view, wherein the partition view records an association between an ID of a current partition and an address of a storage disk; and
An interface configured to provide an external connection;
A computer readable medium configured to store a computer program;
By being connected to the memory, the interface, and the computer readable medium and operating the program,
Obtaining a key in the key-value data and calculating an ID of a confirmed partition corresponding to the key-value data according to the key, wherein the key-value data uniquely corresponds to the value and the value Including the key being, and
Calculating an ID of a current partition corresponding to the ID of the defined partition, wherein each current partition ID corresponds to an ID of a plurality of confirmed partitions; and
Searching the partition view to obtain the address of the storage disk corresponding to the ID of the current partition;
Generating a key-value packet by using the address of the storage disk as a destination address, and transmitting the key-value packet to the storage disk via the interface, wherein the key-value packet is A data processing device comprising: a processor configured to perform the steps of conveying key-value data. The memory is further configured to record the total number L of confirmed partitions, and calculating the confirmed partition ID corresponding to the key value data according to the key comprises:
Performing a hash operation on the key to obtain a hash value of the key, and performing a modulo operation on the hash value according to the total number L of deterministic partitions, wherein a remainder is a value of the deterministic partition; 12. A data processing device according to claim 11, comprising in particular a step used as said ID. The memory is further configured to record a total number T of current partitions, and calculating an ID of a current partition corresponding to the ID of the confirmed partition includes:
Performing a modulo operation on the ID of the defined partition according to the total number T of current partitions, wherein a remainder is used as the ID of the current partition, and the ID of the defined partition is greater than or equal to 0 13. A data processing device according to claim 11 or 12, particularly comprising a step that is an integer.   12. The data processing device according to claim 11, wherein the determined partition is a sub-partition of the current partition and is acquired by dividing the current partition. Prior to obtaining the key in the key value data, the processor
The method further comprising: dividing the data to be written to obtain a value set that includes the value, and generating the key of the value to form the key value data. 12. The data processing device according to 12. A partition management method executed by a controller, wherein the controller performs partition management for storage disks in a cluster, and the cluster includes a plurality of storage disks, the method comprising:
Upon detecting that N new storage disks are about to be added to the cluster, obtaining a number M of current storage disks in the cluster and a total number T of current partitions in the cluster, wherein M, N and T are all natural numbers, steps, and
Determining whether the mathematical relationship between the total number T of current partitions and the total number M + N of storage disks satisfies a first preset condition;
If the relationship satisfies the first preset condition, dividing the at least one current partition so that the total number S of partitions obtained after the division satisfies S>T; and Allocating partitions to the M + N storage disks, the mathematical relationship between the total number S of partitions obtained after the partitioning and the total number M + N of storage disks is a second preset condition And the total number of partitions obtained after the partitioning is not greater than the total number L of deterministic partitions supported by the cluster, and both L and S are natural numbers greater than 1, a partition management method comprising: . The mathematical relationship between the total number T of current partitions and the total number M + N of storage disks satisfies the first preset condition,
17. The method of claim 16, wherein T / (M + N) is less than a first threshold and the first threshold is a natural number.   The method of claim 17, wherein the first threshold is greater than 10 and less than 20. The mathematical relationship between the total number S of partitions obtained after the division and the total number M + N of storage disks satisfies a second preset condition,
17. The method of claim 16, wherein S / (M + N) is greater than a second threshold and refers to the second threshold being a natural number.   The method of claim 19, wherein the second threshold is greater than 25 and less than 50. The ID of the current partition is an integer greater than or equal to 0, and the set of all the partition IDs of the current partition has an initial term of 0, a term number of T, a tolerance of 1, and an arithmetic sequence And
The partition ID of the partition acquired after the division is an integer greater than or equal to 0, and the set of all the partition IDs of the partition acquired after the division is the first term is 0, the number of terms is S, 17. The method of claim 16, wherein the tolerance is 1 and is an arithmetic progression. A partition management device configured to perform partition management on storage disks in a cluster, the cluster including a plurality of storage disks, the device comprising:
Upon detecting that N new storage disks are about to be added to the cluster, it is configured to obtain the number M of current storage disks in the cluster and the total number T of currently existing partitions in the cluster A storage disk detection module, where M, N, and T are all natural numbers;
A first preset condition determination module configured to determine whether a mathematical relationship between the total number T of current partitions and the total number M + N of storage disks satisfies a first preset condition;
If the relationship satisfies the first preset condition, divide at least one of the current partitions so that the total number of partitions acquired after the division is S, and the partition acquired after the division is the M + A partitioning module configured to allocate to N storage disks, wherein the mathematical relationship between the total number S of partitions obtained after the partitioning and the total number M + N of storage disks is a second The total number of partitions obtained after the partitioning is not greater than the total number L of confirmed partitions supported by the cluster, and both L and S are natural numbers greater than 1, A partition management device. That the total number T of current partitions satisfies the first preset condition,
23. The partition management device according to claim 22, wherein T / (M + N) is less than a first threshold value, and the first threshold value is a natural number.   24. The partition management device according to claim 23, wherein the first threshold value is greater than 10 and less than 20. The mathematical relationship between the total number S of partitions obtained after the division and the total number M + N of storage disks satisfies a second preset condition,
23. The partition management device according to claim 22, wherein S / (M + N) is larger than a second threshold value, and indicates that the second threshold value is a natural number.   26. The partition management device according to claim 25, wherein the second threshold value is greater than 25 and less than 50. The ID of the current partition is an integer greater than or equal to 0, and the set of all the partition IDs of the current partition has an initial term of 0, a term number of T, a tolerance of 1, and an arithmetic sequence And
The partition ID of the partition acquired after the division is an integer greater than or equal to 0, and the set of all the partition IDs of the partition acquired after the division is the first term is 0, the number of terms is S, 26. The partition management device according to claim 25, wherein the tolerance is 1, and is an arithmetic progression. A partition management device connected to a cluster and configured to perform partition management for storage disks in the cluster, the cluster including a plurality of storage disks, the partition management device comprising:
A memory configured to store a partition view, wherein the partition view records an association between an ID of a current partition and an address of a storage disk; and
An interface configured to provide an external interface;
A computer readable medium configured to store a computer program;
By being connected to the memory, the interface, and the computer readable medium and operating the program,
When the interface is used to detect that N new storage disks are about to be added to the cluster, the number M of current storage disks in the cluster and the total number T of currently existing partitions in the cluster are calculated. A step of acquiring, where M, N, and T are all natural numbers; and
Determining whether the mathematical relationship between the total number T of current partitions and the total number M + N of storage disks satisfies a first preset condition;
If the relationship satisfies the first preset condition, at least one current partition is divided so that the total number of partitions acquired after the division is S, and the partition acquired after the division is the M Allocating to + N storage disks, the mathematical relationship between the total number S of partitions obtained after the division and the total number M + N of storage disks satisfies a second preset condition, The total number of partitions obtained after splitting is not greater than the total number L of deterministic partitions supported by the cluster, and both L and S are natural numbers greater than 1, a processor configured to perform the steps And a partition management device. That the total number T of current partitions satisfies the first preset condition,
29. The partition management device according to claim 28, wherein T / (M + N) is less than a first threshold value, and indicates that the first threshold value is a natural number.   30. The partition management device according to claim 29, wherein the first threshold value is greater than 10 and less than 20. The mathematical relationship between the total number S of partitions obtained after the division and the total number M + N of storage disks satisfies a second preset condition,
29. The partition management device according to claim 28, wherein S / (M + N) is larger than a second threshold value, and indicates that the second threshold value is a natural number.   32. The partition management device according to claim 31, wherein the second threshold value is greater than 25 and less than 50.

Images