WO2024001126A1 - 一种纠删码融合方法、系统、电子设备及非易失性可读存储介质 - Google Patents

一种纠删码融合方法、系统、电子设备及非易失性可读存储介质 Download PDF

Info

Publication number
WO2024001126A1
WO2024001126A1 PCT/CN2022/142520 CN2022142520W WO2024001126A1 WO 2024001126 A1 WO2024001126 A1 WO 2024001126A1 CN 2022142520 W CN2022142520 W CN 2022142520W WO 2024001126 A1 WO2024001126 A1 WO 2024001126A1
Authority
WO
WIPO (PCT)
Prior art keywords
blocks
data
check code
cabinet
fusion
Prior art date
Application number
PCT/CN2022/142520
Other languages
English (en)
French (fr)
Inventor
吴睿振
陈静静
张永兴
张旭
王小伟
王凛
Original Assignee
苏州元脑智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏州元脑智能科技有限公司 filed Critical 苏州元脑智能科技有限公司
Publication of WO2024001126A1 publication Critical patent/WO2024001126A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • This application relates to the field of data storage technology, and in particular to an erasure code fusion method, system, electronic device and non-volatile readable storage medium.
  • Erasure code is a forward error correction technology in coding theory. It was first used in the field of communications to solve problems such as loss and loss in data transmission. Since erasure coding technology has achieved good results in preventing data loss, it has been introduced into the storage field. Erasure coding can effectively reduce storage overhead while ensuring the same reliability. Therefore, erasure coding technology is widely used in major storage systems and data centers.
  • RS code Random-Solomon Code
  • the RS code is related to two parameters k and r. Given two positive integers k and r, RS code encodes k data blocks into r additional check code blocks.
  • the way in which r check code blocks are encoded based on Vandermond matrix or Cauchy matrix is called RS erasure coding using Vandermond matrix or Cauchy matrix encoding.
  • the common RS erasure code inverse matrix is calculated using the Gaussian elimination method.
  • This general solution is suitable for the inversion of any invertible matrix, but it is not optimized for the characteristics of matrix encoding. Therefore, although the calculation is regular, it will introduce A lot of redundant operations.
  • k data blocks are stored and r check data blocks are added, the probability of errors in a single data block to be recovered is high, and using Gaussian elimination requires (k+r) 3 operations to obtain the required Inverse the matrix and then recover the corresponding data blocks.
  • the purpose of this application is to provide an erasure code fusion method, system, a non-volatile readable storage medium and an electronic device, which can reduce the calculation amount of the erasure code fusion operation.
  • this application provides an erasure code fusion method, which is applied to a distributed storage system.
  • the distributed storage system includes multiple cabinets, and the cabinets include data blocks.
  • the downgrade fusion instruction after receiving the downgrade fusion instruction, it also includes: determining the number of remaining cabinets in the distributed storage system; where the number of remaining cabinets is the number of cabinets in the distributed storage system that do not store the check code block; converting the data The ratio of the number of blocks to the number of remaining cabinets sets a reference threshold; cabinets whose number of stored data blocks is greater than the reference threshold are set as cabinets to be migrated, and the data blocks in the cabinets to be migrated are migrated to free cabinets.
  • set a reference threshold for the ratio of the number of data blocks to the number of remaining cabinets including: determining whether the ratio of the number of data blocks to the number of remaining cabinets is an integer; if so, setting the ratio of the number of data blocks to the number of remaining cabinets. Set as the reference threshold; if not, round up the ratio of the number of data blocks to the number of remaining cabinets, and set the rounded result as the reference threshold.
  • the ratio between the number of data blocks and the number of remaining cabinets is not an integer, it also includes: rounding down the ratio between the number of data blocks and the number of remaining cabinets, and making the number of stored data blocks smaller than the rounded result.
  • the cabinet is set as an idle cabinet.
  • migrating the data blocks in the cabinet to be migrated to the idle cabinet includes: migrating the data blocks in the cabinet to be migrated to the idle cabinet according to the preset constraints; where the preset constraints are: all data blocks are in the same location. Bands.
  • the idle cabinet after migrating the data blocks in the cabinet to be migrated to the idle cabinet, it also includes: adjusting the storage location of the data blocks in the distributed storage system according to the load balancing algorithm so that the data stored in any two cabinets The difference in the number of blocks is less than the preset difference.
  • merging the check code blocks in multiple strips according to the check code block distribution includes: dividing all check code blocks into multiple code block groups according to the check code block distribution, each The values of r corresponding to all check code blocks in the code block group are the same; fuse the check code blocks in the same code block group that are in the same cabinet to obtain a new check code block; merge the check code blocks in the same code block group that are not in the same cabinet The check code blocks are migrated to the same cabinet so that they can be merged in the same cabinet to obtain new check code blocks.
  • fusing the check code blocks in the same code block group and located in the same cabinet includes: fusing the check code blocks in the same code block group and located in the same cabinet according to the preset formula to obtain a new check code block. ; Among them, the default formula is:
  • P r ′ d x,1 ⁇ j 0 +d x,2 ⁇ j 1 +...+d x,m ⁇ j m-1 +(d y,1 ⁇ j 0 +d y,2 ⁇ j 1 +...+d y, m ⁇ j m-1 ) ⁇ j m ;
  • P r ′ is the new check code block, d x, 1 , d x, 2 ..., d x, m is the x
  • the data of the 1st to mth data blocks in the stripe, dy ,1 , dy,2 ..., dy,m are the data of the 1st to mth data blocks in the yth stripe.
  • the check code blocks that are not in the same cabinet in the same code block group to the same cabinet it also includes: determining whether the data amount corresponding to the check code block migration is greater than the preset data amount; if so, determine The downgrade fusion instruction corresponds to the downgrade requirements, and generates a new check code block that meets the downgrade requirements based on the data code block. If not, proceed to the operation of migrating the check code blocks in the same code block group that are not in the same cabinet to the same cabinet.
  • the generation process of the downgrade fusion instruction includes: determining whether the target stripe exists in the distributed storage system; the target stripe is a stripe whose read and write frequency is less than the preset frequency; if so, generating the downgrade fusion instruction.
  • it also includes: moving data blocks to the same stripe by adjusting the stripe address mapping relationship.
  • the check code block distribution after fusing the check code blocks in multiple stripes according to the check code block distribution, it also includes: calculating the current redundancy ratio of the distributed storage system, and uploading the current redundancy ratio to user interface.
  • the distributed storage system includes multiple cabinets, and the cabinets include data blocks.
  • This application also provides a non-volatile readable storage medium on which a computer program is stored. When the computer program is executed, the steps of the above erasure code fusion method are implemented.
  • This application also provides an electronic device, including a memory and a processor.
  • a computer program is stored in the memory.
  • the processor calls the computer program in the memory, the steps of executing the above erasure code fusion method are implemented.
  • the distributed storage system includes multiple cabinets, and the cabinets include data blocks.
  • This application generates check code blocks for all data blocks in the strip according to preset rules.
  • the r-th check code block in the strip can be directly fused. After receiving the downgrade fusion command, the check code block distribution of multiple stripes in the distributed storage system is obtained.
  • the check code blocks with the same r value in different strips can be fused according to the check code block distribution. No need By recalculating the check code block based on the data block, this application can reduce the calculation amount of the erasure code fusion operation.
  • This application also provides an erasure code fusion system, a non-volatile readable storage medium and an electronic device, which have the above beneficial effects and will not be described in detail here.
  • Figure 1 is a flow chart of an erasure code fusion method provided by an embodiment of the present application
  • Figure 2 is a schematic structural diagram of a distributed storage system provided by an embodiment of the present application.
  • Figure 3 is a flow chart of a data block migration method provided by an embodiment of the present application.
  • Figure 4 is a flow chart of a check code block fusion method provided by an embodiment of the present application.
  • Figure 5 is a schematic diagram of a data storage principle under erasure code protection provided by an embodiment of the present application.
  • Figure 6 is a schematic diagram of the conventional erasure code fusion operation principle
  • Figure 7 is a schematic diagram of data storage in a distributed storage system provided by an embodiment of the present application.
  • Figure 8 is a schematic diagram of the first set of check code block fusion results provided by the embodiment of the present application.
  • Figure 9 is a schematic diagram of the second set of check code block fusion results provided by the embodiment of the present application.
  • Figure 10 is a schematic diagram of the third set of check code block fusion results provided by the embodiment of the present application.
  • Figure 11 is a schematic structural diagram of an erasure code fusion system provided by an embodiment of the present application.
  • Figure 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • Figure 13 is a schematic structural diagram of a non-volatile readable storage medium provided by an embodiment of the present application.
  • Figure 1 is a flow chart of an erasure code fusion method provided by an embodiment of the present application.
  • S101 Generate check code blocks for all striped data blocks in the cabinet of the distributed storage system according to preset rules
  • the distributed storage system includes multiple cabinets, and the cabinets include data blocks.
  • Figure 2 is a schematic structural diagram of a distributed storage system provided by an embodiment of the present application; the example in Figure 2 has 4 cabinets, each cabinet is inserted with multiple hard disks, and the hard disks are configured differently based on the operation. Chunk size (chunk size), each chunk is the basic unit of storage operation, and the operation unit that can be seen when operating at the upper level control is the block. As shown in Figure 2, blocks in the same row are in the same strip.
  • Each cabinet will have its own control hardware that connects and controls multiple disks through PCIE and other methods.
  • the cabinets After receiving the requirements from the upper layer, it divides different numbers of blocks into corresponding encoding and decoding.
  • the cabinets are connected through network cables, etc., and then connected to the switch for corresponding command transfer and control.
  • the upper layer of the switch is the host, which decomposes different user needs into different commands for users and issues them respectively for corresponding control.
  • This embodiment can store data blocks in each strip of the distributed storage system, and generate a check code for the data block based on an erasure coding algorithm.
  • the block storing the check code in the cabinet is a check code block.
  • the data of the r-th check code block in d n, m is the data of the m-th data block in the n-th stripe.
  • the addition operations in the above formula are all addition operations in the Galois field.
  • the downgrade fusion instruction is an instruction that controls the distributed storage system to perform a downgrade fusion operation.
  • the above downgrade fusion operation refers to a method of reducing the redundancy ratio in the distributed storage system by merging check code blocks.
  • the redundancy ratio is the ratio of the number of check code blocks to the number of data blocks.
  • the process includes: determining whether there is a target stripe in the distributed storage system; the target stripe is one whose read and write frequency is less than the preset frequency. strip; if so, generate a downgrade fusion instruction.
  • the cabinet location where the r-th check code block in different strips is located can be determined according to the distribution of check code blocks, so that check code blocks with the same r value (ie, check code block sequence number) can be merged.
  • check code block fusion operation Take two strips as an example to illustrate the check code block fusion operation:
  • the j-th check code block in the x-th strip is d x, 1 ⁇ j 0 +d x, 2 ⁇ j 1 +...+d x, m ⁇ j m-1 , y -th strip
  • the jth check code block in is d y, 1 ⁇ j 0 + d y, 2 ⁇ j 1 +...+d y, m ⁇ j m-1 .
  • d x, 1 , d x, 2 ..., d x, m are the data of the 1st to mth data blocks in the x-th stripe, d y, 1 , d y, 2 ...
  • this embodiment can also move data blocks to the same stripe by adjusting the stripe address mapping relationship, so as to manage data in the distributed storage system.
  • the current redundancy ratio of the distributed storage system can also be calculated, and the current redundancy ratio can be calculated. The remaining proportion is uploaded to the user interface.
  • This embodiment generates check code blocks for all data blocks in the strip according to preset rules.
  • the r-th check code blocks in different strips can be directly fused. After receiving the downgrade fusion command, the check code block distribution of multiple stripes in the distributed storage system is obtained. The check code blocks with the same r value in different strips can be fused according to the check code block distribution. No need By recalculating the check code block based on the data block, this embodiment can reduce the calculation amount of the erasure code fusion operation.
  • Figure 3 is a flow chart of a data block migration method provided by an embodiment of the present application. This embodiment can be combined with the embodiment corresponding to Figure 1 to obtain a further implementation manner. After receiving the downgrade fusion After the instruction, you can also perform the following operations:
  • S301 Determine the number of remaining cabinets in the distributed storage system; where the number of remaining cabinets is the number of cabinets in the distributed storage system that do not store check code blocks;
  • the process of setting the reference threshold for the ratio of the number of data blocks to the number of remaining cabinets includes: determining whether the ratio of the number of data blocks to the number of remaining cabinets is an integer; if so, adding the sum of the number of data blocks and the number of remaining cabinets. The ratio of the number of remaining cabinets is set as the reference threshold; if not, the ratio of the number of data blocks and the number of remaining cabinets is rounded up, and the rounded result is set as the reference threshold.
  • the ratio between the number of data blocks and the number of remaining cabinets is not an integer, the ratio between the number of data blocks and the number of remaining cabinets can also be rounded down, and the number of stored data blocks is smaller than the cabinet that is rounded down. Set as a free cabinet.
  • this embodiment can migrate the data blocks in the cabinet to be migrated to the idle cabinet in the following manner: migrate the data blocks in the cabinet to be migrated to the idle cabinet according to the preset constraint conditions; where the preset constraint conditions are: all The data blocks are in the same stripe.
  • the storage location of the data blocks in the distributed storage system can also be adjusted according to the load balancing algorithm so that the number of data blocks stored in any two cabinets The difference is less than the preset difference value.
  • Figure 4 is a flow chart of a verification code block fusion method provided by an embodiment of the present application. This embodiment can be combined with the embodiment corresponding to Figure 1 to obtain a further implementation manner.
  • the above-mentioned verification Code block fusion refers to fusing the check codes in multiple check code blocks to obtain a new check code block.
  • the embodiment corresponding to Figure 4 may include the following steps:
  • S403 Migrate the check code blocks in the same code block group that are not in the same cabinet to the same cabinet so that they can be merged in the same cabinet to obtain new check code blocks.
  • the check code blocks in the same code block group in the same cabinet can be fused in the following way: follow the preset formula to fuse the check code blocks in the same code block group
  • Check code block fusion specifically refers to the fusion of data in the check code block.
  • the above j represents the jth parity block in the stripe.
  • the check code blocks in the same code block group in the same cabinet can be fused in the following way: follow the preset formula to fuse the check code blocks in the same code block group in the same cabinet.
  • the check code blocks of the same cabinet are fused to obtain a new check code block; among which, the preset formula is:
  • P r ′ d x1 , 1 ⁇ j 0 +d x1, 2 ⁇ j 1 +...+d x1, m ⁇ j m-1 +(d x2, 1 ⁇ j 0 +d x2, 2 ⁇ j 1 +...+d x2,m ⁇ j m- 1 ) ⁇ j m +(d x3,1 ⁇ j 0 +d x3,2 ⁇ j 1 +...+d x3,m ⁇ j m-1 ) ⁇ j m+1 +...+(d xz, 1 ⁇ j 0 +d xz, 2 ⁇ j 1 +...+d xz, m ⁇ j m-1 ) ⁇ j m-2+z ;
  • P r ′ is the new check code block, d x1, 1 , d x1, 2 ..., d x1, m is the data of the 1st to mth data blocks in the x1th strip, d x2, 1
  • the data amount corresponding to the check code block migration is greater than the preset data amount; if so, determine the downgrade fusion Downgrade requirements corresponding to the instructions, and generate new check code blocks that meet the downgrade requirements based on the data code blocks. If not, proceed to the operation of migrating the check code blocks in the same code block group that are not in the same cabinet to the same cabinet.
  • the conventional erasure coding encoding process is as follows:
  • the k*k matrix in the above content corresponds to k original data blocks, and the r*k matrix in the lower part corresponds to the encoding matrix.
  • the original data D 1 to D k By multiplying the original data D 1 to D k , the newly added P 1 to P are obtained.
  • r is the r verification data obtained by encoding.
  • the inverse matrix of the corresponding matrix of the remaining data is multiplied by the data, and the original data blocks D 1 to D k are obtained.
  • the RS erasure code recovery method is as follows:
  • erasure coding is to construct an invertible coding matrix to generate check data, and its inverse matrix can be calculated to restore the original data.
  • Common RS erasure codes use the Cauchy matrix or Vandermonde matrix introduced above. The advantage is that the obtained matrix is definitely reversible, any of its sub-matrices are also reversible, and the size of the matrix can be easily expanded.
  • FIG. 5 is a schematic diagram of a data storage principle under erasure code protection provided by an embodiment of the present application.
  • the data storage in the first row is a and b respectively.
  • User requirements require that it can verify and correct any two errors. Therefore, based on the erasure algorithm, the 3rd and 4th cabinets here respectively store the 1st data generated based on Vandermond. and the second check code block, based on the Vandermond algorithm, are a+b and a+2b respectively.
  • the addition operations here are all additions in the Galois field, and are actually performed as XOR operations.
  • the second and third rows show the storage situation when the user data is not full and only requires an error check and correction.
  • the check codes generated based on the Vandermond algorithm are c+d and e+f respectively.
  • the specific operation is the same as above.
  • the fourth line has only one data, so the check code is g.
  • Figure 5 is an example illustrating the difference in different situations. The specific scenario is far more complicated than that.
  • the redundancy ratio at this time is 1/2, that is to say, through data protection at this time, each Two user data will generate one verification data. Although this kind of protection effect is good, the cost is high.
  • the redundancy ratio is reduced to 1/4, that is, c, d, e, and f will appear through the common
  • the check code at this time should be c+d+e+f.
  • the involved data blocks need to be shifted first, and then the check code part is recalculated.
  • Figure 6 is a schematic diagram of the conventional erasure code fusion operation principle.
  • the operation shown in Figure 6 involves a lot of operations between cabinets, and the encoding needs to be redone, resulting in a large speed loss. The loss is even greater when it involves the recalculation of multiple check codes.
  • Figure 7 is a schematic diagram of data storage in a distributed storage system provided by an embodiment of the present application.
  • the scenario shown in Figure 7 is the storage situation of the same user, so that the data of the same user can be downgraded and merged. .
  • the example in Figure 7 illustrates a practical situation. At this time, it is assumed that there are more cabinets and more data storage for the same user. This involves different scenarios that need to be processed by this application, which can be divided into the first group, the second group and the The third group has three types.
  • d is the user data to be encoded
  • m is the amount of user data to be encoded
  • r is the number of checks based on the redundancy ratio requirement.
  • This embodiment can determine the adjustment status of the data block and the adjustment status of the check code, and perform operations according to scenarios.
  • the data block adjustment situation is divided into two categories: whether the data block needs to be adjusted.
  • check code adjustment there are two types: calculation using check code blocks and calculation using data blocks. Different situations are illustrated by the first group, the second group, and the third group in the above figure. The processing solutions of this application under different situations are explained below.
  • the check code blocks are in the same cabinets 4, 5, and 6, and the data blocks are evenly distributed in the remaining cabinets. Therefore, the scenario at this time is a scenario where there is no need to adjust the data block and the check code block completely overlaps.
  • pu and pd are the upward and downward rounding values of p respectively.
  • the number of data blocks in each cabinet in the first group is less than or equal to 2 (that is, pu) and greater than or equal to 1 (that is, pd), so the data blocks in the first group do not need to be migrated.
  • check code blocks in the first group completely overlap.
  • the corresponding data conditions of the first group after downgraded fusion are: a ⁇ f.
  • the check codes p 1 ⁇ p after downgraded fusion are obtained. 3 conditions, you need to have:
  • Figure 8 is a schematic diagram of the first set of check code block fusion results provided by the embodiment of the present application.
  • no user data needs to be migrated in data fusion; in terms of check code fusion, only Calculation is performed based on p 1, 2 , p 2, 2 , p 2, 3 , and the calculations are all inside the cabinet.
  • SoC control communication protocols such as PCIe are used for transmission, and the calculation can be completed. It does not involve transmission between cabinets, and the data calculation requirements are small, so a great speed improvement can be achieved.
  • FIG. 9 is a schematic diagram of the second set of check code block fusion results provided by the embodiment of the present application.
  • the fused check code blocks are g+h+(i+j) and g+2h+4(1+2j).
  • the check blocks in the third group do not overlap at all, so m+n and m+2n in cabinets 7 and 8 are migrated respectively, and then the fusion operation is performed. The final result is shown in Figure 10, which is provided by the embodiment of this application. Schematic diagram of the third set of check code block fusion results.
  • the migrated check code blocks are k+l+(m+n) and k+2l+4(m+2n).
  • All operations in the above embodiment are Galois field operations, and the data migration arrangement therein only needs to meet the requirements of different load balancing algorithms.
  • This embodiment changes the data encoding method, the judgment of check blocks and data blocks, and the different operation problems of data blocks and check blocks in different situations.
  • this embodiment uses as much in-cabinet computing as possible to reduce inter-cabinet network work and improve the efficiency of downgrading and integration.
  • This embodiment proposes an efficient and simple operation scheme based on user requirements or scenario requirements for downgrading and fusion operations under distributed storage.
  • the formed distributed storage group can determine the status of data blocks and check blocks and perform fusion and downgrade operations.
  • This fusion and downgrade reduces data transmission between cabinets and makes the calculation of check blocks only require It can be based on the original check block of the fused data, which simplifies the calculation and makes more use of the PCIe high-speed data transmission protocol in the cabinet, thereby reducing unnecessary transmission between cabinets, reducing the CPU load, and reducing channel occupancy, thereby achieving Optimization of speed.
  • Figure 11 is a schematic structural diagram of an erasure code fusion system provided by an embodiment of the present application. This system can be applied to a distributed storage system.
  • the distributed storage system includes multiple cabinets, and the cabinets include data blocks.
  • Erasure code fusion system including:
  • the check generation module 1101 is used to generate check code blocks for all striped data blocks in the cabinet of the distributed storage system according to preset rules;
  • the data of the r-th check code block in the strip, d n, m is the data of the m-th data block in the n-th stripe;
  • the distribution acquisition module 1102 is used to obtain the distribution of check code blocks of multiple stripes in the distributed storage system if a downgrade fusion instruction is received;
  • the fusion module 1103 is used to fuse the check code blocks in multiple strips according to the distribution of check code blocks.
  • This embodiment generates check code blocks for all data blocks in the strip according to preset rules.
  • the r-th check code blocks in different strips can be directly fused. After receiving the downgrade fusion command, the check code block distribution of multiple stripes in the distributed storage system is obtained. The check code blocks with the same r value in different strips can be fused according to the check code block distribution. No need By recalculating the check code block based on the data block, this embodiment can reduce the calculation amount of the erasure code fusion operation.
  • the data block migration module is used to determine the number of remaining cabinets in the distributed storage system after receiving the downgrade fusion command; where the number of remaining cabinets is the number of cabinets in the distributed storage system that do not store the check code block; it is also used It is used to set a reference threshold for the ratio of the number of data blocks to the number of remaining cabinets; it is also used to set cabinets whose number of stored data blocks is greater than the reference threshold as cabinets to be migrated, and to migrate the data blocks in the cabinets to be migrated to free cabinets.
  • the process of the data block migration module setting the reference threshold for the ratio of the number of data blocks to the number of remaining cabinets includes: determining whether the ratio of the number of data blocks to the number of remaining cabinets is an integer; if so, adding the number of data blocks to the number of remaining cabinets.
  • the ratio of the number of cabinets is set as the reference threshold; if not, the ratio of the number of data blocks and the number of remaining cabinets is rounded up, and the rounded result is set as the reference threshold.
  • the idle cabinet determination module is used to round down the ratio of the number of data blocks to the number of remaining cabinets if the ratio between the number of data blocks and the number of remaining cabinets is not an integer, and the number of stored data blocks will be less than the rounded result.
  • the cabinet is set as an idle cabinet.
  • the process of the data block migration module migrating the data blocks in the cabinet to be migrated to the idle cabinet includes: migrating the data blocks in the cabinet to be migrated to the idle cabinet according to the preset constraints; where the preset constraints are: all The data blocks are in the same stripe.
  • the load balancing module is used to adjust the storage location of the data blocks in the distributed storage system according to the load balancing algorithm after migrating the data blocks in the cabinet to be migrated to the idle cabinet, so that the data blocks stored in any two cabinets
  • the quantity difference is less than the preset difference.
  • the process of merging the check code blocks in multiple strips according to the distribution of check code blocks by the fusion module 1103 includes: dividing all check code blocks into multiple code block groups according to the distribution of check code blocks. , the values of r corresponding to all check code blocks in each code block group are the same; fuse the check code blocks in the same code block group in the same cabinet to obtain a new check code block; merge different code blocks in the same code block group
  • the check code blocks in the same cabinet are migrated to the same cabinet so that they can be merged in the same cabinet to obtain new check code blocks.
  • the process of merging the check code blocks in the same cabinet in the same code block group by the fusion module 1103 includes: merging the check code blocks in the same code block group in the same cabinet according to a preset formula to obtain a new check code block.
  • Pr′ is the new check code block, dx, 1, dx, 2..., dx, m is the 1st to mth in the x-th strip
  • the data of the data block, dy, 1, dy, 2..., dy, m is the data of the 1st to mth data blocks in the yth stripe.
  • the judgment module is used to judge whether the data volume corresponding to the migration of the verification code blocks is greater than the preset data volume before migrating the verification code blocks that are not in the same cabinet in the same code block group to the same cabinet; if so, determine the downgrade fusion
  • the corresponding downgrade requirements of the instruction are generated, and a new check code block that meets the downgrade requirements is generated based on the data code block; if not, the operation of migrating the check code blocks in the same code block group that are not in the same cabinet to the same cabinet is entered.
  • the generation process of the downgrade fusion instruction includes: determining whether the target stripe exists in the distributed storage system; the target stripe is a stripe whose read and write frequency is less than the preset frequency; if so, generating the downgrade fusion instruction.
  • the stripe adjustment module is used to move data blocks to the same stripe by adjusting the stripe address mapping relationship.
  • the feedback module is used to calculate the current redundancy ratio of the distributed storage system after fusing the check code blocks in multiple strips according to the distribution of the check code blocks, and upload the current redundancy ratio to the user interface. .
  • This embodiment proposes an efficient and simple operation scheme based on user requirements or scenario requirements for downgrading and fusion operations under distributed storage.
  • the formed distributed storage group can determine the status of data blocks and check blocks and perform fusion and downgrade operations.
  • This fusion and downgrade reduces data transmission between cabinets and makes the calculation of check blocks only require It can be based on the original check block of the fused data, which simplifies the calculation and makes more use of the PCIe high-speed data transmission protocol in the cabinet, thereby reducing unnecessary transmission between cabinets, reducing the CPU load, and reducing channel occupancy, thereby achieving Optimization of speed.
  • FIG. 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 12, the electronic device includes:
  • Communication interface 1201 can interact with other devices such as network devices;
  • the processor 1202 is connected to the communication interface 1201 to implement information interaction with other devices, and is used to execute the erasure code fusion method provided by one or more of the above technical solutions when running a computer program.
  • the computer program is stored on memory 1203.
  • bus system 1204. is used to implement connection communication between these components.
  • the bus system 1204 also includes a power bus, a control bus and a status signal bus.
  • the various buses are labeled bus system 1204 in FIG. 12 .
  • the non-volatile readable storage medium may include: semiconductor memory chip, USB flash drive, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code.
  • Figure 13 is a schematic structural diagram of a non-volatile readable storage medium provided by an embodiment of the present application.
  • the non-volatile readable storage medium can be a semiconductor memory chip, specifically including a decoding driver, a storage matrix, a read and write circuit, address lines, data lines, chip select lines and read/write control lines.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Read Only Memory (AREA)
  • Storage Device Security (AREA)

Abstract

本申请公开了一种纠删码融合方法、系统、电子设备及非易失性可读存储介质。所属的技术领域为数据存储技术领域。所述纠删码融合方法应用于分布式存储系统，所述分布式存储系统包括多个机柜，所述机柜中包括数据块，所述纠删码融合方法，包括：按照预设规则在所述分布式存储系统的机柜中生成所有条带的数据块的校验码块；若接收到降级融合指令，则获取所述分布式存储系统中多个条带的校验码块分布情况；根据所述校验码块分布情况对多个所述条带中的校验码块进行融合，本申请能够降低纠删码融合操作的计算量。

Description

一种纠删码融合方法、系统、电子设备及非易失性可读存储介质
相关申请的交叉引用
本申请要求于2022年6月28日提交中国专利局，申请号为202210738362.5，申请名称为″一种纠删码融合方法、系统、电子设备及存储介质″的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据存储技术领域，特别涉及一种纠删码融合方法、系统、电子设备及非易失性可读存储介质。
背景技术
面对海量数据的存储要求，分布式存储以其成本低廉，可扩展性好等优势逐渐取代了统一存储的主导地位，在理论研究和实际应用方面得到了越来越多的关注。分布式存储系统多以廉价的磁盘作为存储节点，每个存储节点的可靠性往往不会很高；另一方面，分布式存储系统通常包含很多的节点，由于软硬件故障，人为失误等原因，系统常常发生节点失效的情况。为了提高分布式存储系统的数据可靠性，保证数据收集节点能以很高的概率实现原始文件的重构，需要在存储原始数据的基础上，额外存储一定数量的冗余，使得在出现部分节点失效的情况下，系统仍然可以正常运行，数据收集节点仍然可以对原始文件实现解码恢复。同时，为了维持系统的可靠性，需要对失效的节点及时进行修复，因此良好的节点修复机制十分重要。
纠删码(Erasure Code)属于编码理论中的一种前向纠错技术，最早应用于通信领域以解决数据传输中的丢失与损耗这类问题。由于纠删码技术在防止数据丢失取得了较好的效果，因此被引入存储领域。纠删码可以在保证相同可靠性的前提下有效地降低存储开销，因此纠删码技术被广泛地应用于各大存储系统以及数据中心。
纠删码的种类众多，在实际存储系统中较常见的有应用在分布式环境下的RS码(Reed-Solomon Code，里德所罗门码)。RS码与两个参数k和r相关。给定两个正整数k和r，RS码将k个数据块编码为r个额外的校验码块。而r个校验码块基于范德蒙矩阵或柯西矩阵进行编码的方式就称为利用范德蒙矩阵或柯西矩阵编码的RS纠删码。
常见的RS纠删码逆矩阵的计算采用的是高斯消元法，这种通用解法适用于任何可逆矩阵的求逆，但是没有针对矩阵编码的特性进行优化，因此虽然计算规律化，却会引入大量冗余运算。当存储k个数据块，添加r个校验数据块的情况下，需要恢复的单个数据块错误的几率较高，而使用高斯消元需要(k+r) 3次运算，才能得到所需要的逆矩阵，然后恢复相应的数据块。
在分布式存储下，多个机柜通过网络连接，通过上层主机的控制，实现存储及其安全备份功能。实际应用中，存在用户对使用的降级融合需求，而此时因为分布式存储的空间使用问题，机柜内部的硬件计算 和外部网络连接速度的问题，不同的操作方式会有不同的速度表现，传统的操作方式为基于标准的编码流程重新组建纠删组，这样的方式速度很慢，损耗很大。
因此，如何降低纠删码融合操作的计算量是本领域技术人员目前需要解决的技术问题。
发明内容
本申请的目的是提供一种纠删码融合方法、系统、一种非易失性可读存储介质及一种电子设备，能够降低纠删码融合操作的计算量。
为解决上述技术问题，本申请提供一种纠删码融合方法，应用于分布式存储系统，分布式存储系统包括多个机柜，机柜中包括数据块，纠删码融合方法，包括：按照预设规则在分布式存储系统的机柜中生成所有条带的数据块的校验码块；其中，预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据；若接收到降级融合指令，则获取分布式存储系统中多个条带的校验码块分布情况；根据校验码块分布情况对多个条带中的校验码块进行融合。
可选的，在接收到降级融合指令之后，还包括：确定分布式存储系统中的剩余机柜数量；其中，剩余机柜数量为分布式存储系统中未存储校验码块的机柜的数量；将数据块的数量和剩余机柜数量的比值设置参考阈值；将存储的数据块数量大于参考阈值的机柜设置为待迁移机柜，并将待迁移机柜中的数据块迁移至空闲机柜。
可选的，将数据块的数量和剩余机柜数量的比值设置参考阈值，包括：判断数据块的数量和剩余机柜数量的比值是否为整数；若是，则将数据块的数量和剩余机柜数量的比值设置为参考阈值；若否，则对数据块的数量和剩余机柜数量的比值向上取整，将向上取整结果设置为参考阈值。
可选的，若数据块的数量和剩余机柜数量的比值不为整数，还包括：对数据块的数量和剩余机柜数量的比值向下取整，将存储的数据块数量小于向下取整结果的机柜设置为空闲机柜。
可选的，将待迁移机柜中的数据块迁移至空闲机柜，包括：按照预设约束条件将待迁移机柜中的数据块迁移至空闲机柜；其中，预设约束条件为：所有数据块处于同一条带。
可选的，在将待迁移机柜中的数据块迁移至空闲机柜之后，还包括：按照负载均衡算法对分布式存储系统中的数据块存储位置进行调整，以使任意两个机柜中存储的数据块数量差小于预设差值。
可选的，根据校验码块分布情况对多个条带中的校验码块进行融合，包括：根据校验码块分布情况将所有校验码块划分为多个码块组，每一码块组中所有校验码块对应的r的数值相同；将同一码块组中处于同一机柜的校验码块进行融合，得到新校验码块；将同一码块组中不处于同一机柜的校验码块迁移至同一机柜，以便在同一机柜中进行融合，得到新校验码块。
可选的，将同一码块组中处于同一机柜的校验码块进行融合，包括：按照预设公式将同一码块组中处于同一机柜的校验码块进行融合，得到新校验码块；其中，预设公式为：
P r′＝d x，1×j 0+d x，2×j 1+...+d x，m×j m-1+(d y，1×j 0+d y，2×j 1+...+d y，m×j m-1)×j m；P r′为新校验码块，d x，1、d x，2...、d x，m为第x个条带中第1至第m个数据块的数据，d y，1、d y，2...、d y，m为第y个条带中第1至第m个数据块的数据。
可选的，在将同一码块组中不处于同一机柜的校验码块迁移至同一机柜之前，还包括：判断校验码块迁移对应的数据量是否大于预设数据量；若是，则确定降级融合指令对应的降级需求，并根据数据码块生成符合降级需求的新校验码块。若否，则进入将同一码块组中不处于同一机柜的校验码块迁移至同一机柜的操作。
可选的，降级融合指令的生成过程包括：判断分布式存储系统中是否存在目标条带；目标条带为读写频率小于预设频率的条带；若是，则生成降级融合指令。
可选的，还包括：通过调整条带地址映射关系将数据块移动至同一条带中。
可选的，在根据校验码块分布情况对多个条带中的校验码块进行融合之后，还包括：计算分布式存储系统当前的冗余比例，并将当前的冗余比例上传至用户界面。
本申请还提供了一种纠删码融合系统，应用于分布式存储系统，分布式存储系统包括多个机柜，机柜中包括数据块，纠删码融合系统，包括：校验生成模块，用于按照预设规则在分布式存储系统的机柜中生成所有条带的数据块的校验码块；其中，预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据；分布情况获取模块，用于若接收到降级融合指令，则获取分布式存储系统中多个条带的校验码块分布情况；融合模块，用于根据校验码块分布情况对多个条带中的校验码块进行融合。
本申请还提供了一种非易失性可读存储介质，其上存储有计算机程序，计算机程序执行时实现上述纠删码融合方法执行的步骤。
本申请还提供了一种电子设备，包括存储器和处理器，存储器中存储有计算机程序，处理器调用存储器中的计算机程序时实现上述纠删码融合方法执行的步骤。
本申请提供了一种纠删码融合方法，应用于分布式存储系统，分布式存储系统包括多个机柜，机柜中包括数据块，纠删码融合方法，包括：按照预设规则在分布式存储系统的机柜中生成所有条带的数据块的校验码块；其中，预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据；若接收到降级融合指令，则获取分布式存储系统中多个条带的校验码块分布情况；根据校验码块分布情况对多个条带中的校验码块进行融合。
本申请按照预设规则生成所有条带中数据块的校验码块，上述预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据，不同条带中的第r个校验码块可以直接融合。在收到降级融合指令之后获取分布式存储系统中多个条带的校验码块分布情况，可以根据校验码块分布情况将不同条带中r值相同的校验码块进行融合，无需根据数据块重 新计算校验码块，本申请能够降低纠删码融合操作的计算量。本申请同时还提供了一种纠删码融合系统、一种非易失性可读存储介质和一种电子设备，具有上述有益效果，在此不再赘述。
附图说明
为了更清楚地说明本申请实施例，下面将对实施例中所需要使用的附图做简单的介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请实施例所提供的一种纠删码融合方法的流程图；
图2为本申请实施例所提供的一种分布式存储系统的结构示意图；
图3为本申请实施例所提供的一种数据块迁移方法的流程图；
图4为本申请实施例所提供的一种校验码块融合方法的流程图；
图5为本申请实施例所提供的一种纠删码保护下的数据存储原理示意图；
图6为常规的纠删码融合操作原理示意图；
图7为本申请实施例所提供的一种分布式存储系统的数据存储示意；
图8为本申请实施例所提供的第一组校验码块融合结果示意图；
图9为本申请实施例所提供的第二组校验码块融合结果示意图；
图10为本申请实施例所提供的第三组校验码块融合结果示意图；
图11为本申请实施例所提供的一种纠删码融合系统的结构示意图；
图12为本申请实施例所提供的一种电子设备的结构示意图；
图13为本申请实施例所提供的一种非易失性可读存储介质的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
下面请参见图1，图1为本申请实施例所提供的一种纠删码融合方法的流程图。
具体步骤可以包括：
S101：按照预设规则在分布式存储系统的机柜中生成所有条带的数据块的校验码块；
其中，本实施例可以应用于分布式存储系统，分布式存储系统包括多个机柜，机柜中包括数据块。请参见图2，图2为本申请实施例所提供的一种分布式存储系统的结构示意图；图2中的举例有4个机柜，每个机柜插有多块硬盘，硬盘基于操作设置不同的块大小(chunk size)，每个块是存储操作的基本单位，在上层控制时进行操作能看到的操作单位即为块。如图2中，同一行的块处于同一条带。每个机柜会有一 块自己的控制硬件，通过PCIE等方式连接并控制多块磁盘，接收上层的需求后，划分不同数量的块进行相应的编解码。机柜与机柜之间通过网线等方式连接，然后通过插在交换机上进行相应的命令转移和控制。交换机上层是主机，面向用户将不同的用户需求拆解为不同的命令，分别下发进行相应的控制。
本实施例可以在分布式存储系统的每一条带中存储数据块，并基于纠删码算法生成数据块的校验码，机柜中存储校验码的块为校验码块。上述预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据。上式中的加法运算都是伽罗华域下的加法运算。
S102：若接收到降级融合指令，则获取分布式存储系统中多个条带的校验码块分布情况；
其中，降级融合指令为控制分布式存储系统执行降级融合操作的指令，上述降级融合操作指：通过合并校验码块将分布式存储系统中的冗余比降低的方式。冗余比为校验码块数量与数据块数量之比。在接收到降级融合指令后，可以获取分布式存储系统中多个条带的校验码块分布情况，上述校验码块分布情况包括各个校验码块所在的机柜位置。
作为一种可行的实施方式，在本步骤之前还可以存在生成降级融合指令的操作，其过程包括：判断分布式存储系统中是否存在目标条带；目标条带为读写频率小于预设频率的条带；若是，则生成降级融合指令。
S103：根据校验码块分布情况对多个条带中的校验码块进行融合。
其中，根据校验码块分布情况可以确定不同条带中的第r个校验码块所在的机柜位置，以便将r值(即校验码块序号)相同的校验码块进行融合。以两个条带为例，说明校验码块融合操作：
第x个条带中的第j个校验码块为d x，1×j 0+d x，2×j 1+...+d x，m×j m-1，第 y个条带中的第j个校验码块为d y，1×j 0+d y，2×j 1+...+d y，m×j m-1，按照以下公式进行校验码块融合：P r′＝d x，1×j 0+d x，2×j 1+...+d x，m×j m- 1+(d y，1×j 0+d y，2×j 1+...+d y，m×j m-1)×j m；P r′为融合后的新校验码块。上式中，d x，1、d x，2...、d x，m为第x个条带中第1至第m个数据块的数据，d y，1、d y，2...、d y，m为第y个条带中第1至第m个数据块的数据。由于本实施例中按照预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1生成校验码块，可以使r值相同的校验码块直接融合，无需根据数据块重新计算校验码块。
作为一种可行的实施方式，本实施例还可以通过调整条带地址映射关系将数据块移动至同一条带中，以便对分布式存储系统中的数据进行管理。
作为另一种可行的实施方式，在根据校验码块分布情况对多个条带中的校验码块进行融合之后，还可以计算分布式存储系统当前的冗余比例，并将当前的冗余比例上传至用户界面。
本实施例按照预设规则生成所有条带中数据块的校验码块，上述预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据，不同条带中的第r个校验码块可以直接融合。在收到降级融合指令之后获取分布式存储系统中多个条 带的校验码块分布情况，可以根据校验码块分布情况将不同条带中r值相同的校验码块进行融合，无需根据数据块重新计算校验码块，本实施例能够降低纠删码融合操作的计算量。
请参见图3，图3为本申请实施例所提供的一种数据块迁移方法的流程图，可以将本实施例与图1对应的实施例相结合得到进一步的实施方式，在接收到降级融合指令之后，还可以执行以下操作：
S301：确定分布式存储系统中的剩余机柜数量；其中，剩余机柜数量为分布式存储系统中未存储校验码块的机柜的数量；
S302：将数据块的数量和剩余机柜数量的比值设置参考阈值；
S303：将存储的数据块数量大于参考阈值的机柜设置为待迁移机柜，并将待迁移机柜中的数据块迁移至空闲机柜。
作为一种可行的实施方式，将数据块的数量和剩余机柜数量的比值设置参考阈值的过程包括：判断数据块的数量和剩余机柜数量的比值是否为整数；若是，则将数据块的数量和剩余机柜数量的比值设置为参考阈值；若否，则对数据块的数量和剩余机柜数量的比值向上取整，将向上取整结果设置为参考阈值。
进一步的，若数据块的数量和剩余机柜数量的比值不为整数，还可以对数据块的数量和剩余机柜数量的比值向下取整，将存储的数据块数量小于向下取整结果的机柜设置为空闲机柜。
进一步的，本实施例可以通过以下方式将待迁移机柜中的数据块迁移至空闲机柜：按照预设约束条件将待迁移机柜中的数据块迁移至空闲机柜；其中，预设约束条件为：所有数据块处于同一条带。
进一步的，在将待迁移机柜中的数据块迁移至空闲机柜之后，还可以按照负载均衡算法对分布式存储系统中的数据块存储位置进行调整，以使任意两个机柜中存储的数据块数量差小于预设差值。
请参见图4，图4为本申请实施例所提供的一种校验码块融合方法的流程图，可以将本实施例与图1对应的实施例相结合得到进一步的实施方式，上述校验码块融合指：将多个校验码块中的校验码进行融合得到一个新的校验码块。图4对应的实施例可以包括以下步骤：
S401：根据校验码块分布情况将所有校验码块划分为多个码块组，每一码块组中所有校验码块对应的r的数值相同；
S402：将同一码块组中处于同一机柜的校验码块进行融合，得到新校验码块；
S403：将同一码块组中不处于同一机柜的校验码块迁移至同一机柜，以便在同一机柜中进行融合，得到新校验码块。
以处于同一机柜的两个条带进行校验码块融合为例，可以通过以下方式将同一码块组中处于同一机柜的校验码块进行融合：按照预设公式将同一码块组中处于同一机柜的校验码块进行融合，得到新校验码块；其中，预设公式为：P r′＝d x，1×j 0+d x，2×j 1+...+d x，m×j m-1+(d y，1×j 0+d y，2×j 1+...+d y，m×j m-1)×j m；P r′为新校验码块，d x，1、d x，2...、d x，m为第x个条带中第1至第m个数据块的数据，d y，1、d y，2...、d y，m为第y个条带中第1至第m个数据块的数据。校验码块融合具体指校验码块中的数据的融合。上述j表示条带中第j个校验 块。
以处于同一机柜的多个条带进行校验码块融合为例，可以通过以下方式将同一码块组中处于同一机柜的校验码块进行融合：按照预设公式将同一码块组中处于同一机柜的校验码块进行融合，得到新校验码块；其中，预设公式为：
P r′＝d x1，1×j 0+d x1，2×j 1+...+d x1，m×j m-1+(d x2，1×j 0+d x2，2×j 1+...+d x2，m×j m- 1)×j m+(d x3，1×j 0+d x3，2×j 1+...+d x3，m×j m-1)×j m+1+...+(d xz，1×j 0+d xz，2×j 1+...+d xz，m×j m-1)×j m-2+z；P r′为新校验码块，d x1，1、d x1，2...、d x1，m为第x1个条带中第1至第m个数据块的数据，d x2，1、d x2，2...、d x2，m为第x2个条带中第1至第m个数据块的数据，d x3，1、d x3，2...、d x3，m为第x3个条带中第1至第m个数据块的数据，d xz，1、d xz，2...、d xz，m为第xz个条带中第1至第m个数据块的数据。
进一步的，在将同一码块组中不处于同一机柜的校验码块迁移至同一机柜之前，还可以判断校验码块迁移对应的数据量是否大于预设数据量；若是，则确定降级融合指令对应的降级需求，并根据数据码块生成符合降级需求的新校验码块。若否，则进入将同一码块组中不处于同一机柜的校验码块迁移至同一机柜的操作。
下面通过在实际应用中的实施例说明上述实施例描述的流程。
常规的纠删码编码过程如下：
基于范德蒙矩阵的RS纠删码：
Figure PCTCN2022142520-appb-000001
基于柯西矩阵的RS纠删码：
Figure PCTCN2022142520-appb-000002
以上内容中的k*k矩阵对应的就是k个原始数据块，下部分的r*k矩阵对应的就是编码矩阵，通过与原始数据D 1到D k相乘，得到新添加的P 1到P r就是编码所得到的r个校验数据。当其中任意做多r个数据在传输中出错或丢失，需要纠错时，即用剩余数据对应矩阵的逆矩阵与数据相乘，即会得到原始数据块D 1到D k。
以D 1到D r数据丢失，进行解码为例，RS纠删码恢复数据方式如下所示：
Figure PCTCN2022142520-appb-000003
可知纠删码的核心概念是构建一个可逆的编码矩阵用以产生校验数据，其逆矩阵可经过计算恢复原始数据。常见的RS纠删码使用的是上面介绍的柯西矩阵或范德蒙矩阵，这样的优势是所得到的矩阵肯定可逆，其任意子矩阵也都可逆，并且矩阵的大小扩充简单。
分布式存储下通常有多个机柜，每个机柜中有多个磁盘，主机面向的会是多个用户的不同需求，而不同用户的数据都有数据安全性的需求，因此分布式存储的数据需要通过纠删码对其进行保护。请参见图5，图5为本申请实施例所提供的一种纠删码保护下的数据存储原理示意图。
如图5中所示，举例了几种分布式存储下对数据进行保护的类型。第一行的数据存储分别为a和b，用户需求需要其可以校验纠正任意两个错误，因此基于纠删算法，这里的第3机柜和第4机柜分别存储了基于范德蒙所生成的第1和第2个校验码块，基于范德蒙算法，分别是a+b和a+2b，这里的加法运算都是伽罗华域下的加法，实际表现为异或运算。第二和第三行表现的是用户数据没有存满，且只需要一个错误校验纠正的存储情况，此时校验码分别基于范德蒙算法生成的校验码分别是c+d和e+f，具体运算同上。第四行只有一个数据，因此校验码为g。图5为举例说明不同情况下的区别，具体场景远比其复杂。
在图5中的存储情况下，此时假设第二行和第三行的存储场景是同一个用户的需求，此时的冗余比为1/2，即是说此时通过数据保护，每两个用户数据，会生成一个校验数据，这样的保护效果虽然好，但是成本较高。此时假设用户数据的安全性需求下降了，有可能会出现主诉降低冗余比，进行降级融合的需求，将冗余比下降为1/4，即是出现c，d，e，f通过共同生成一个校验码的情况，基于范德蒙算法，此时的校验码应该为c+d+e+f。对于传统的操作方式而言，首先需要对涉及的数据块进行移位，然后重新计算校验码部分。对于上例中的情况，传统操作情况如图6所示，图6为常规的纠删码融合操作原理示意图。图6所示的操作涉及了较多机柜间的操作，且编码需要重新做，速度损耗较大，当涉及多个校验码的重新计算时损耗更大。
为解决上述相关技术中存在的问题，本申请实施例提供了一种分布式存储下RS纠删码的高效融合方案，该方案适用于各种情况，减少机柜间的数据传输，降低了运算复杂度。请参见图7，图7为本申请实施例所提供的一种分布式存储系统的数据存储示意图，图7所示的场景为同一个用户的存储情况，以便对同一个用户的数据进行降级融合。如图7的举例说明了一种实际情况，此时假设机柜较多，同一个用户的数据存储也较多，涉及本申请需要处理的不同场景，可以分为如上第一组、第二组和第三组三种。综合的 划分场景，可以分为以下两种情况：情况1、是否需要调整数据块以达到负载均衡；情况2、降级融合涉及运算的校验码块的关系。本实施例可以判断两种情况，然后进行不同的操作，即可达到最优化降级融合。
本实施例基于范德蒙算法生成校验码的方式如下：
p 1＝d 1+d 2+...+d m
p 2＝d 1*2 0+d 2*2 1+...+d m2 m-1
...
p r＝d 1*r 0+d 2*r 1+...+d mr m-1
上式中d为待编码用户数据，m为待编码的用户数据量，r为基于冗余比需求的校验数量。
本实施例可以判断数据块调整情况及校验码调整情况，并分场景进行操作。数据块调整情况下分为是否需要调整数据块两类。校验码调整情况下分为使用校验码块计算和使用数据块进行计算两类。不同情况通过上图中第一组、第二组和第三组举例说明，下面分别对不同情况下的本申请处理方案进行说明。
第一组：
以第一组情况举例说明，此时校验码块在相同的机柜4、5、6内，数据块平均分布在了剩余的机柜内。因此此时的场景为：无需调整数据块以及校验码块完全重合的场景。
对于是否满足无需调整数据块的情况，判断方法为：设除了校验码存放的机柜外剩余数据块可存放的机柜数为m，总数据块为k，则在第一组中m＝5，k＝6。
Figure PCTCN2022142520-appb-000004
Figure PCTCN2022142520-appb-000005
其中pu和pd为p分别的向上取整和向下取整值。
则当每机柜内现在数据块数量w满足pd≤w≤pu时，则判定为无需调整数块的情况，此时数据块完全不做操作，保持原位置。
在第一组中各机柜的数据块数量均小于或等于2(即pu)且大于或等于1(即pd)，故第一组中的数据块无需迁移。
进一步的，第一组中校验码块完全重合，降级融合后对应的第一组的数据情况为：a～f，基于相同的范德蒙算法，为得到降级融合后的校验码p 1～p 3条件，则需要有：
p 1＝a+b+c+d+e+f
p 2＝a+2b+4c+8d+16e+32f
p 3＝a+3b+9c+27d+81e+243f
以上公式中的加法运算均为伽罗华域运算。
而我们现有第一行和第二行分别的校验码p 1.1～p 1.3和p 2.1～p 2.3关系为：
p 1，1＝a+b+c
p 1，2＝a+2b+4c
p 1，3＝a+3b+9c
p 2，1＝d+e+f
p 2，2＝d+2e+4f
p 2，3＝d+3e+9f
当校验码块在同一个机柜内重合时，使用校验码块直接进行融合计算，具体方法如下：
p 1＝p 1，1+1 rp 2，1
p 2＝p 1，2+2 rp 2，2
p 3＝p 1，3+3 rp 2，3
上式中的r为用户选择冗余比后所确定的校验码块数量，在第一组中，每三个用户数据，会生成3个校验码块进行保护，则相应的r＝3。针对对应的p1，p2，p3选择1，2，3依次类推作为底，进行r的幂次运算，然后相加即可得到所需要的融合后的结果。
以上述第一组情况为例，则这里的计算为：
p 1＝p 1，1+1 rp 2，1＝a+b+c+d+e+f
p 2＝p 1，2+2 rp 2，2＝a+2b+4c+2 3(d+2e+4f)
p 3＝p 1，3+3 rp 2，3＝a+3b+9c+3 3(d+3e+9f)
请参见图8，图8为本申请实施例所提供的第一组校验码块融合结果示意图，此时数据融合在数据迁移方面没有任何用户数据需要迁移；在校验码融合上，只需要基于p 1，2，p 2，2，p 2，3进行计算，而计算都在机柜内部，基于SoC控制下，利用PCIe等通信协议进行传输，完成计算即可。不涉及机柜间的传输，且数据计算需求量小，可以获得很大的速度提升。
第二组：
本实施例可以通过以下方式判断数据块调整情况及校验码调整情况，此时基于上文公式，m＝5，k＝4，pu＝1，pw＝0，根据各个机柜的实际存储情况，可以将机柜2中的数据块i迁移至机柜5。基于负载均衡算法还可以将数据块j迁移至机柜6。迁移针对同条带下任意空白机柜进行数据迁移皆可为准，则这里将i和j分别迁移至机柜5和机柜6。
第二组中的机柜4的校验码块有重合，但因为机柜4的校验码块更新需要用到的是机柜5的i+2j，机柜3需要用到机柜4的i+j，因此机柜4和5都需要对校验码块进行迁移。请参见图9，图9为本申请实施 例所提供的第二组校验码块融合结果示意图。融合后的校验码块为g+h+(i+j)和g+2h+4(1+2j)。
第三组：
本实施例可以通过以下方式判断数据块调整情况及校验码调整情况，此时基于上文公式，m＝4，k＝4，参考阈值为1，此时无须进行数据迁移，则各个数据保持机柜内原始位置。其中m和n在条带6，可以在静默状态下更新地址，调整条带地址映射关系，使其在同一条带内即可。第三组中校验块完全没有重合，则机柜7和8内的m+n和m+2n分别进行迁移，然后进行融合运算，最终得到结果如图10，图10为本申请实施例所提供的第三组校验码块融合结果示意图。迁移后的校验码块为k+l+(m+n)以及k+2l+4(m+2n)。
根据图10可知，这里m+n和m+2n分别进行了跨机柜的迁移，而校验码块m和n其实并非涉及数据迁移，仅仅改变主机host中对应的地址即可。
上述实施例中所有运算都是伽罗华域运算，其中的数据迁移排布方式满足不同的负载均衡算法的要求即可。
本实施例改变了数据编码方式、校验块和数据块的判断情况，不同情况下针对数据块和校验块的场景的不同操作问题，当需要融合时的数据迁移判断方式和迁移方式，当需要融合时的校验块运算方式。本实施例针对不同场景，尽量多利用机柜内运算，减少机柜间网络工作的方案，提高降级融合的效率。
本实施例基于分布式存储下用户需求，或场景需求进行降级融合操作的情况，提出了一种高效简易的操作方案。通过改变编码方式，所组建的分布式存储组可以在判断数据块和校验块的情况，进行融合降级的操作，该融合降级减少了机柜间的数据传输，并且使校验块的计算只需要基于被融合数据的原本校验块即可，简化了运算，更多的利用机柜内的PCIe的高速数据传输协议，而降低机柜间的不必要传输，降低CPU的负载，减少通道占用，从而达到速度的最优化。
请参见图11，图11为本申请实施例所提供的一种纠删码融合系统的结构示意图，该系统可以应用于分布式存储系统，分布式存储系统包括多个机柜，机柜中包括数据块，纠删码融合系统，包括：
校验生成模块1101，用于按照预设规则在分布式存储系统的机柜中生成所有条带的数据块的校验码块；
其中，预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据；
分布情况获取模块1102，用于若接收到降级融合指令，则获取分布式存储系统中多个条带的校验码块分布情况；
融合模块1103，用于根据校验码块分布情况对多个条带中的校验码块进行融合。
本实施例按照预设规则生成所有条带中数据块的校验码块，上述预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数 据，不同条带中的第r个校验码块可以直接融合。在收到降级融合指令之后获取分布式存储系统中多个条带的校验码块分布情况，可以根据校验码块分布情况将不同条带中r值相同的校验码块进行融合，无需根据数据块重新计算校验码块，本实施例能够降低纠删码融合操作的计算量。
进一步的，还包括：
数据块迁移模块，用于在接收到降级融合指令之后，确定分布式存储系统中的剩余机柜数量；其中，剩余机柜数量为分布式存储系统中未存储校验码块的机柜的数量；还用于将数据块的数量和剩余机柜数量的比值设置参考阈值；还用于将存储的数据块数量大于参考阈值的机柜设置为待迁移机柜，并将待迁移机柜中的数据块迁移至空闲机柜。
进一步的，数据块迁移模块将数据块的数量和剩余机柜数量的比值设置参考阈值的过程包括：判断数据块的数量和剩余机柜数量的比值是否为整数；若是，则将数据块的数量和剩余机柜数量的比值设置为参考阈值；若否，则对数据块的数量和剩余机柜数量的比值向上取整，将向上取整结果设置为参考阈值。
进一步的，还包括：
空闲机柜确定模块，用于若数据块的数量和剩余机柜数量的比值不为整数，对数据块的数量和剩余机柜数量的比值向下取整，将存储的数据块数量小于向下取整结果的机柜设置为空闲机柜。
进一步的，数据块迁移模块将待迁移机柜中的数据块迁移至空闲机柜的过程包括：按照预设约束条件将待迁移机柜中的数据块迁移至空闲机柜；其中，预设约束条件为：所有数据块处于同一条带。
进一步的，还包括：
负载均衡模块，用于在将待迁移机柜中的数据块迁移至空闲机柜之后，按照负载均衡算法对分布式存储系统中的数据块存储位置进行调整，以使任意两个机柜中存储的数据块数量差小于预设差值。
进一步的，融合模块1103根据校验码块分布情况对多个条带中的校验码块进行融合的过程包括：根据校验码块分布情况将所有校验码块划分为多个码块组，每一码块组中所有校验码块对应的r的数值相同；将同一码块组中处于同一机柜的校验码块进行融合，得到新校验码块；将同一码块组中不处于同一机柜的校验码块迁移至同一机柜，以便在同一机柜中进行融合，得到新校验码块。
进一步的，融合模块1103将同一码块组中处于同一机柜的校验码块进行融合的过程包括：按照预设公式将同一码块组中处于同一机柜的校验码块进行融合，得到新校验码块；其中，预设公式为：Pr′＝dx，1×j0+dx，2×j1+...+dx，m×jm-1+(dy，1×j0+dy，2×j1+...+dy，m×jm-1)×jm；Pr′为新校验码块，dx，1、dx，2...、dx，m为第x个条带中第1至第m个数据块的数据，dy，1、dy，2...、dy，m为第y个条带中第1至第m个数据块的数据。
进一步的，还包括：
判断模块，用于在将同一码块组中不处于同一机柜的校验码块迁移至同一机柜之前，判断校验码块迁移对应的数据量是否大于预设数据量；若是，则确定降级融合指令对应的降级需求，并根据数据码块生成 符合降级需求的新校验码块；若否，则进入将同一码块组中不处于同一机柜的校验码块迁移至同一机柜的操作。
进一步的，降级融合指令的生成过程包括：判断分布式存储系统中是否存在目标条带；目标条带为读写频率小于预设频率的条带；若是，则生成降级融合指令。
进一步的，还包括：
条带调整模块，用于通过调整条带地址映射关系将数据块移动至同一条带中。
进一步的，还包括：
反馈模块，用于在根据校验码块分布情况对多个条带中的校验码块进行融合之后，计算分布式存储系统当前的冗余比例，并将当前的冗余比例上传至用户界面。
本实施例基于分布式存储下用户需求，或场景需求进行降级融合操作的情况，提出了一种高效简易的操作方案。通过改变编码方式，所组建的分布式存储组可以在判断数据块和校验块的情况，进行融合降级的操作，该融合降级减少了机柜间的数据传输，并且使校验块的计算只需要基于被融合数据的原本校验块即可，简化了运算，更多的利用机柜内的PCIe的高速数据传输协议，而降低机柜间的不必要传输，降低CPU的负载，减少通道占用，从而达到速度的最优化。
由于系统部分的实施例与方法部分的实施例相互对应，因此系统部分的实施例请参见方法部分的实施例的描述，这里暂不赘述。
本申请还提供了一种电子设备，可以包括存储器和处理器，存储器中存有计算机程序，处理器调用存储器中的计算机程序时，可以实现上述实施例所提供的步骤。当然电子设备还可以包括各种网络接口，电源等组件。图12为本申请实施例所提供的一种电子设备的结构示意图，如图12所示，电子设备包括：
通信接口1201，能够与其它设备比如网络设备等进行信息交互；
处理器1202，与通信接口1201连接，以实现与其它设备进行信息交互，用于运行计算机程序时，执行上述一个或多个技术方案提供的纠删码融合方法。而计算机程序存储在存储器1203上。
当然，实际应用时，电子设备中的各个组件通过总线系统1204耦合在一起。可理解，总线系统1204用于实现这些组件之间的连接通信。总线系统1204除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图12中将各种总线都标为总线系统1204。
本申请还提供了一种非易失性可读存储介质，其上存有计算机程序，该计算机程序被执行时可以实现上述实施例所提供的步骤。该非易失性可读存储介质可以包括：半导体存储芯片、U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。图13为本申请实施例所提供的一种非易失性可读存储介质的结构示意图，该非易失性可读存储介质可以为半导体存储芯片，具体包括译码驱动、存储矩阵、读写电路、地址线、数据线、片选线和读/写控制线。
说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的系统而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以对本申请进行若干改进和修饰，这些改进和修饰也落入本申请权利要求的保护范围内。
还需要说明的是，在本说明书中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语″包括″、″包含″或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的状况下，由语句″包括一个......″限定的要素，并不排除在包括要素的过程、方法、物品或者设备中还存在另外的相同要素。

Claims (20)

  1. 一种纠删码融合方法，其特征在于，应用于分布式存储系统，所述分布式存储系统包括多个机柜，所述机柜中包括数据块，所述纠删码融合方法，包括：
    按照预设规则在所述分布式存储系统的机柜中生成所有条带的数据块的校验码块；其中，所述预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据；
    若接收到降级融合指令，则获取所述分布式存储系统中多个条带的校验码块分布情况；
    根据所述校验码块分布情况对多个所述条带中的校验码块进行融合。
  2. 根据权利要求1所述纠删码融合方法，其特征在于，在接收到降级融合指令之后，还包括：
    确定所述分布式存储系统中的剩余机柜数量；其中，所述剩余机柜数量为所述分布式存储系统中未存储校验码块的机柜的数量；
    将所述数据块的数量和所述剩余机柜数量的比值设置参考阈值；
    将存储的数据块数量大于所述参考阈值的机柜设置为待迁移机柜，并将所述待迁移机柜中的数据块迁移至空闲机柜。
  3. 根据权利要求2所述纠删码融合方法，其特征在于，将所述数据块的数量和所述剩余机柜数量的比值设置参考阈值，包括：
    判断所述数据块的数量和所述剩余机柜数量的比值是否为整数；
    若所述数据块的数量和所述剩余机柜数量的比值是整数，则将所述数据块的数量和所述剩余机柜数量的比值设置为所述参考阈值；
    若所述数据块的数量和所述剩余机柜数量的比值不是整数，则对所述数据块的数量和所述剩余机柜数量的比值向上取整，将向上取整结果设置为所述参考阈值。
  4. 根据权利要求2所述纠删码融合方法，其特征在于，若所述数据块的数量和所述剩余机柜数量的比值不为整数，还包括：
    对所述数据块的数量和所述剩余机柜数量的比值向下取整，将存储的数据块数量小于向下取整结果的机柜设置为所述空闲机柜。
  5. 根据权利要求2所述纠删码融合方法，其特征在于，将所述待迁移机柜中的数据块迁移至空闲机柜，包括：
    按照预设约束条件将所述待迁移机柜中的数据块迁移至空闲机柜；其中，所述预设约束条件为：所有所述数据块处于同一条带。
  6. 根据权利要求2所述纠删码融合方法，其特征在于，在将所述待迁移机柜中的数据块迁移至空闲机柜之后，还包括：
    按照负载均衡算法对所述分布式存储系统中的数据块存储位置进行调整，以使任意两个机柜中存储的数据块数量差小于预设差值。
  7. 根据权利要求1所述纠删码融合方法，其特征在于，根据所述校验码块分布情况对多个所述条带中的校验码块进行融合，包括：
    根据所述校验码块分布情况将所有所述校验码块划分为多个码块组，每一码块组中所有校验码块对应的 r的数值相同；
    将同一所述码块组中处于同一机柜的校验码块进行融合，得到新校验码块；
    将同一所述码块组中不处于同一机柜的校验码块迁移至同一机柜，以便在同一机柜中进行融合，得到新校验码块。
  8. 根据权利要求7所述纠删码融合方法，其特征在于，将同一所述码块组中处于同一机柜的校验码块进行融合，包括：
    按照预设公式将同一所述码块组中处于同一机柜的校验码块进行融合，得到新校验码块；其中，所述预设公式为：
    P r′＝d x，1×j 0+d x，2×j 1+...+d x，m×j m-1+(d y，1×j 0+d y，2×j 1+...+d y，m×j m-1)×j m；
    P r′为新校验码块，d x，1、d x，2...、d x，m为第x个条带中第1至第m个数据块的数据，d y，1、d y，2...、d y，m为第y个条带中第1至第m个数据块的数据。
  9. 根据权利要求7所述纠删码融合方法，其特征在于，在将同一所述码块组中不处于同一机柜的校验码块迁移至同一机柜之前，还包括：
    判断校验码块迁移对应的数据量是否大于预设数据量；
    若所述校验码块迁移对应的数据量大于所述预设数据量，则确定所述降级融合指令对应的降级需求，并根据数据码块生成符合所述降级需求的新校验码块；
    若所述校验码块迁移对应的数据量小于等于所述预设数据量，则进入将同一所述码块组中不处于同一机柜的校验码块迁移至同一机柜的操作。
  10. 根据权利要求1所述纠删码融合方法，其特征在于，所述降级融合指令的生成过程包括：
    判断所述分布式存储系统中是否存在目标条带；所述目标条带为读写频率小于预设频率的条带；
    若所述分布式存储系统中存在目标条带，则生成所述降级融合指令。
  11. 根据权利要求1所述纠删码融合方法，其特征在于，还包括：
    通过调整条带地址映射关系将所述数据块移动至同一条带中。
  12. 根据权利要求1所述纠删码融合方法，其特征在于，在根据所述校验码块分布情况对多个所述条带中的校验码块进行融合之后，还包括：
    计算所述分布式存储系统当前的冗余比例，并将所述当前的冗余比例上传至用户界面。
  13. 根据权利要求1所述纠删码融合方法，其特征在于，所述机柜中设置有控制硬件，所述控制硬件连接并控制多块磁盘，所述控制硬件在接收到用户需求的情况下，对所述硬盘划分不同数量的数据块进行编解码。
  14. 根据权利要求1所述纠删码融合方法，其特征在于，所述降级融合操作是指通过合并校验码块将所述分布式存储系统中的冗余比降低，其中，所述冗余比为校验码块数量与数据块数量之比。
  15. 根据权利要求1所述纠删码融合方法，其特征在于，在校验码块在相同的多个机柜内，数据块平均分布在剩余的机柜内的情况下，在数据融块合时，数据块保持原位置，在校验码块融合时，在机柜内进行融合计算。
  16. 根据权利要求1所述纠删码融合方法，其特征在于，在不同校验码块在同一条带的情况下，在静默状态下更新校验块的地址，并调整条带地址映射关系。
  17. 根据权利要求1所述纠删码融合方法，其特征在于，在机柜内的校验块完全重合的情况下，对机柜内的对校验块进行融合计算；在机柜内的校验块没有完全重合的情况下，对机柜内的校验块进行跨迁移，并进行融合计算。
  18. 一种纠删码融合系统，其特征在于，应用于分布式存储系统，所述分布式存储系统包括多个机柜，所述机柜中包括数据块，所述纠删码融合系统，包括：
    校验生成模块，用于按照预设规则在所述分布式存储系统的机柜中生成所有条带的数据块的校验码块；
    其中，所述预设规则为P n，r＝d n，1×r 0+d n，2×r 1+...+d n，m×r m-1，P n，r为第n个条带中第r个校验码块的数据，d n，m为第n个条带中第m个数据块的数据；
    分布情况获取模块，用于若接收到降级融合指令，则获取所述分布式存储系统中多个条带的校验码块分布情况；
    融合模块，用于根据所述校验码块分布情况对多个所述条带中的校验码块进行融合。
  19. 一种电子设备，其特征在于，包括存储器和处理器，所述存储器中存储有计算机程序，所述处理器调用所述存储器中的计算机程序时实现如权利要求1至17任一项所述纠删码融合方法的步骤。
  20. 一种非易失性可读存储介质，其特征在于，所述非易失性可读存储介质中存储有计算机可执行指令，所述计算机可执行指令被处理器加载并执行时，实现如权利要求1至17任一项所述纠删码融合方法的步骤。
PCT/CN2022/142520 2022-06-28 2022-12-27 一种纠删码融合方法、系统、电子设备及非易失性可读存储介质 WO2024001126A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210738362.5A CN114816837B (zh) 2022-06-28 2022-06-28 一种纠删码融合方法、系统、电子设备及存储介质
CN202210738362.5 2022-06-28

Publications (1)

Publication Number Publication Date
WO2024001126A1 true WO2024001126A1 (zh) 2024-01-04

Family

ID=82522616

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/142520 WO2024001126A1 (zh) 2022-06-28 2022-12-27 一种纠删码融合方法、系统、电子设备及非易失性可读存储介质

Country Status (2)

Country Link
CN (1) CN114816837B (zh)
WO (1) WO2024001126A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114816837B (zh) * 2022-06-28 2022-12-02 苏州浪潮智能科技有限公司 一种纠删码融合方法、系统、电子设备及存储介质
CN114995770B (zh) * 2022-08-02 2022-12-27 苏州浪潮智能科技有限公司 一种数据处理方法、装置、设备、系统及可读存储介质
CN116312724B (zh) * 2023-05-16 2023-08-15 苏州浪潮智能科技有限公司 一种适用于两校验编码存储系统数据存储方法和装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200241781A1 (en) * 2019-01-29 2020-07-30 Dell Products L.P. Method and system for inline deduplication using erasure coding
CN112860475A (zh) * 2021-02-04 2021-05-28 山东云海国创云计算装备产业创新中心有限公司 基于rs纠删码的校验块恢复方法、装置、系统及介质
CN113918378A (zh) * 2020-07-10 2022-01-11 华为技术有限公司 数据存储方法、存储系统、存储设备及存储介质
CN114816837A (zh) * 2022-06-28 2022-07-29 苏州浪潮智能科技有限公司 一种纠删码融合方法、系统、电子设备及存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10635533B2 (en) * 2018-07-30 2020-04-28 EMC IP Holding Company LLC Efficient computation of parity data in storage system implementing data striping
CN110442535B (zh) * 2019-07-19 2021-06-11 华中科技大学 提高分布式固态盘键值缓存系统可靠性的方法及系统
CN114153651B (zh) * 2022-02-09 2022-04-29 苏州浪潮智能科技有限公司 一种数据编码方法、装置、设备及介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200241781A1 (en) * 2019-01-29 2020-07-30 Dell Products L.P. Method and system for inline deduplication using erasure coding
CN113918378A (zh) * 2020-07-10 2022-01-11 华为技术有限公司 数据存储方法、存储系统、存储设备及存储介质
CN112860475A (zh) * 2021-02-04 2021-05-28 山东云海国创云计算装备产业创新中心有限公司 基于rs纠删码的校验块恢复方法、装置、系统及介质
CN114816837A (zh) * 2022-06-28 2022-07-29 苏州浪潮智能科技有限公司 一种纠删码融合方法、系统、电子设备及存储介质

Also Published As

Publication number Publication date
CN114816837B (zh) 2022-12-02
CN114816837A (zh) 2022-07-29

Similar Documents

Publication Publication Date Title
WO2024001126A1 (zh) 一种纠删码融合方法、系统、电子设备及非易失性可读存储介质
US10416906B2 (en) Rebalancing efficiency with optimal logical insertion locations
US20170192898A1 (en) Intelligent slice caching
US10120596B2 (en) Adaptive extra write issuance within a dispersed storage network (DSN)
US8601347B1 (en) Flash memory device and storage control method
US20140359348A1 (en) Adjusting dispersed storage network traffic due to rebuilding
EP3015986B1 (en) Access method and device for message-type memory module
CN112764661B (zh) 用于管理存储系统的方法、设备和计算机程序产品
US20140156719A1 (en) Utilizing data object storage tracking in a dispersed storage network
CN110058953B (zh) 用于改变存储系统的类型的方法、设备和存储介质
WO2018171111A1 (zh) 多容错性的mds阵列码编码以及修复方法
EP3364541B1 (en) Storage controller, data processing chip, and data processing method
US20170123915A1 (en) Methods and systems for repurposing system-level over provisioned space into a temporary hot spare
WO2024001112A1 (zh) 一种存储服务器的数据迁移方法、系统、设备及非易失性可读存储介质
US20230333931A1 (en) Dynamic storage of encoded data slices in multiple vaults
US10310763B2 (en) Forming a distributed storage network memory without namespace aware distributed storage units
CN112732168B (zh) 用于管理存储系统的方法、设备和计算机程序产品
CN116450048A (zh) Raid6磁盘阵列降级为raid5磁盘阵列的方法
CN116501553A (zh) 数据恢复方法、装置、系统、电子设备及存储介质
CN111506450B (zh) 用于数据处理的方法、设备和计算机程序产品
CN112000278B (zh) 一种热数据存储的自适应局部重构码设计方法及云存储系统
WO2022007225A1 (zh) 数据存储方法、存储系统、存储设备及存储介质
US10031809B2 (en) Efficient method for rebuilding a set of encoded data slices
WO2024051114A1 (zh) 一种数据转存方法、系统、设备及非易失性可读存储介质
WO2023184843A1 (zh) 一种raid编解码方法、装置、设备及可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22949202

Country of ref document: EP

Kind code of ref document: A1