WO2020238736A1 - 一种生成解码矩阵的方法、解码方法和对应装置 - Google Patents

一种生成解码矩阵的方法、解码方法和对应装置 Download PDF

Info

Publication number
WO2020238736A1
WO2020238736A1 PCT/CN2020/091466 CN2020091466W WO2020238736A1 WO 2020238736 A1 WO2020238736 A1 WO 2020238736A1 CN 2020091466 W CN2020091466 W CN 2020091466W WO 2020238736 A1 WO2020238736 A1 WO 2020238736A1
Authority
WO
WIPO (PCT)
Prior art keywords
matrix
pcm
data block
transformation
column
Prior art date
Application number
PCT/CN2020/091466
Other languages
English (en)
French (fr)
Inventor
董元元
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2020238736A1 publication Critical patent/WO2020238736A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
    • G06F11/1012—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices using codes or arrangements adapted for a specific type of error
    • G06F11/1032—Simple parity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present invention relates to the technical field of coding error tolerance, in particular to a method for generating a decoding matrix, a decoding method and a corresponding device.
  • the current big data center is also undergoing technological changes.
  • the frequency of hardware failures and software failures is also increasing.
  • the reliability of the storage system is particularly important.
  • the cost and reliability of data storage are both considerations when designing distributed systems. factor.
  • the erasure code can minimize the storage overhead of the system while ensuring the same data reliability.
  • erasure codes are commonly used for storage in distributed systems in the industry. Compared with multi-copy storage methods, this method can save nearly half of the storage space while ensuring the same fault tolerance.
  • this method also has the problem of consuming a lot of computing resources when disk reconstruction and downgrading read and write; on the other hand, a storage method that is different from multiple copies can directly read the copy on another disk when one disk fails. After a data disk fails, the erasure code needs to read an additional check disk and restore the original data through calculation. This process is called degraded read and write. The speed of calculation directly affects the speed of system read and write. At the same time, it affects the response time of the distributed system.
  • the RaidFile in the traditional file system only supports the writing of data in Buffer mode.
  • RaidFile cannot meet the writing needs of users. Therefore, when a disk fails, in order to respond to user requests and restore data, the system often needs to quickly perform degraded read/write and disk reconstruction, which causes problems in the allocation of computing resources and acceleration of decoding algorithms.
  • the present application provides a method for generating a decoding matrix, a decoding method, and a corresponding device, aiming at a distributed system environment, and optimizing downgraded read and write costs without affecting the system's fault tolerance and storage costs.
  • the present invention provides a method for generating a decoding matrix applied to a distributed system, including:
  • the column HS corresponding to the readable data block obtained by removing the unit failure matrix HL from the transformed PCM matrix is used as a decoding matrix
  • the parity check PCM matrix includes a check block matrix and a unit matrix
  • the unit failure matrix HL is a column corresponding to a failed data block in the transformed PCM matrix.
  • performing matrix transformation on the constructed parity check PCM matrix to separate the unit failure matrix HL includes:
  • the check block matrix of, the last m column is the identity matrix
  • Matrix transformation is performed on the PCM matrix, so that the column corresponding to the failed data block is a unit matrix, and the columns corresponding to the failed data block are separated to form a unit failure matrix HL.
  • the method before performing matrix transformation on the PCM matrix, the method further includes:
  • the method before performing matrix transformation on the PCM matrix, the method further includes:
  • performing matrix transformation on the PCM matrix so that the column corresponding to the invalid data block is the identity matrix includes:
  • the column corresponding to the invalid data block is marked as ⁇ a1, a2,..., ar ⁇ ;
  • changing the element of the i-th row to 1 through row transformation includes:
  • changing all the remaining elements in the ai column except the element in the i-th row to 0 through row transformation includes:
  • the element x in the ai column of the other row is multiplied with the i-th row element, and then this row is subtracted.
  • the present invention provides a decoding method, including:
  • the decoding matrix is a parity check PCM matrix, performing matrix transformation to separate the column HS corresponding to the readable data block;
  • the parity check PCM matrix includes a check block matrix and an identity matrix
  • the present invention provides a device for generating a decoding matrix, including:
  • the transformation module is set to perform matrix transformation on the constructed parity check PCM matrix to separate the unit failure matrix HL;
  • An extraction module configured to remove the transformed PCM matrix from the unit failure matrix HL and obtain the column HS corresponding to the readable data block as a decoding matrix
  • the parity check PCM matrix includes a check block matrix and a unit matrix
  • the unit failure matrix HL is a column corresponding to a failed data block in the transformed PCM matrix.
  • the transformation module includes:
  • the separation unit is configured to perform matrix transformation on the PCM matrix so that the column corresponding to the failed data block is a unit matrix, and separates the columns corresponding to the failed data block to form a unit failure matrix HL.
  • the device further includes: a marking module configured to mark the column corresponding to the invalid data block in the PCM matrix.
  • the device further includes:
  • the judgment module is configured to judge whether the column corresponding to the invalid data block is a unit matrix, and when it is not a unit matrix, the separation unit is enabled; when the unit matrix is current, the separation unit is not enabled.
  • the present invention provides a decoding device, including:
  • the decoding module is configured to obtain the original data block by multiplying the decoding matrix by the sum of the original data block and the check data block as 0;
  • the decoding matrix is a parity check PCM matrix, performing matrix transformation to separate the column HS corresponding to the readable data block;
  • the parity check PCM matrix includes a check block matrix and an identity matrix.
  • this application optimizes the matrix operation in the decoding process to reduce the number of multiplication operations and accelerate the operation speed, thereby reducing the response time of the distributed system.
  • the encoding and decoding speed is also maximized to meet the response time requirements in different scenarios of distributed systems.
  • This application optimizes the decoding matrix to accelerate the matrix operation method during decoding, that is, it can be applied to any erasure code that encodes and decodes through matrix operation, and does not need to change the encoding itself, has wide applicability, and guarantees the user’s Timeliness of request response and reliability of data storage.
  • Fig. 1 is a flowchart of a method for generating a decoding matrix according to an embodiment of the present invention
  • Figure 2 is a schematic diagram of generating a decoding matrix in the related art
  • Fig. 3 is a schematic structural diagram of an apparatus for generating a decoding matrix according to an embodiment of the present invention
  • FIG. 4 is a schematic structural diagram of an apparatus for generating a decoding matrix according to an embodiment of the present invention.
  • FIG. 5 is a schematic structural diagram of an apparatus for generating a decoding matrix according to an embodiment of the present invention.
  • Fig. 6 is a flowchart of a decoding process in a distributed system environment according to an embodiment of the present invention.
  • the distributed system includes multiple data partitions, and each data partition includes one or more data disks for storing data.
  • Each data disk may include one or more processors (CPU), input/ Output interface, network interface and memory (memory).
  • the memory may include non-permanent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash memory
  • Memory is an example of computer readable media.
  • the memory may include one or more modules.
  • Computer-readable media include permanent and non-permanent, removable and non-removable storage media, and information storage can be realized by any method or technology.
  • the information can be computer-readable instructions, data structures, program modules, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices.
  • PRAM phase change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • ROM read only memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other memory technology
  • CD-ROM compact disc
  • an embodiment of the present invention provides a method for generating a decoding matrix, which is applied to a distributed system, and includes:
  • the column HS corresponding to the readable data block obtained by removing the unit failure matrix HL from the transformed PCM matrix is used as a decoding matrix
  • the parity check PCM matrix includes a check block matrix and a unit matrix
  • the unit failure matrix HL is a column corresponding to a failed data block in the transformed PCM matrix.
  • step S101 performing matrix transformation on the constructed parity check PCM matrix to separate the unit failure matrix HL includes:
  • the check block matrix of, the last m column is the identity matrix
  • Matrix transformation is performed on the PCM matrix, so that the column corresponding to the failed data block is a unit matrix, and the columns corresponding to the failed data block are separated to form a unit failure matrix HL.
  • PCM Parity Check Matrix
  • the result of multiplying the PCM matrix with the vector composed of the data block and the check block is exactly 0, so the linear relationship between the data block and each check block is described.
  • Each column of each matrix corresponds to a corresponding data block .
  • HL represents the matrix composed of the columns corresponding to the failed data blocks
  • HS represents the readable data blocks corresponding to the Columns, and put them on both sides of the equal sign, and multiply the corresponding data blocks respectively to form the relationship as shown in Figure 2.
  • the inversion of the HL matrix and the multiplication with the HS matrix are very computationally expensive. Therefore, the process of generating the decoding matrix from the PCM matrix in the embodiment of the present invention is optimized.
  • the separated HL is a unit matrix, which can reduce the inversion operation of HL and the multiplication operation of the inverse matrix of the HL matrix and the HS matrix.
  • the embodiment of the present invention uses the PCM matrix to generate the decoding matrix, and optimizes the calculation process of generating the decoding matrix, further reduces the consumption of computing resources and calculation time, to ensure the speed of disk reconstruction and degraded reading and writing when the data block fails. The reliability of the data and the response time of the system.
  • the method before performing matrix transformation on the PCM matrix in step S102, the method further includes: marking the column corresponding to the invalid data block in the PCM matrix;
  • the method before performing matrix transformation on the PCM matrix in step S102, the method further includes:
  • performing matrix transformation on the PCM matrix in step S102 so that the column corresponding to the invalid data block is the identity matrix includes:
  • the column corresponding to the invalid data block is marked as ⁇ a1, a2,..., ar ⁇ ;
  • the columns of the PCM matrix corresponding to the invalid data blocks are respectively labeled as ⁇ a1, a2,..., ar ⁇ .
  • the column ai corresponding to the invalid data can be obtained.
  • Perform the above operations on i [1,r] respectively, and the HL matrix can be turned into an identity matrix.
  • the matrix row transformation does not affect the establishment of the equation.
  • the HL matrix can be easily transformed into an identity matrix through row transformation, which greatly reduces the number of calculation steps.
  • making the element of the i-th row become 1 through row transformation includes:
  • changing all remaining elements in column ai except for the element in the i-th row to 0 through row transformation includes:
  • the element x in the ai column of the other row is multiplied with the i-th row element, and then this row is subtracted.
  • the use of matrix transformation reduces the operations of a matrix inversion and a matrix multiplication in the process of generating the decoding matrix.
  • This scheme can be directly added to the generation of the decoding matrix without changing the original erasure coding scheme.
  • the size of the parity check PCM matrix is m ⁇ n to illustrate the process of generating the decoding matrix:
  • the check block matrix of, the last m column is the identity matrix
  • the column HS corresponding to the readable data block obtained by removing the invalidation matrix HL from the transformed PCM matrix is used as the decoding matrix.
  • the method before performing matrix transformation on the PCM matrix, the method further includes:
  • the method before performing matrix transformation on the PCM matrix, the method further includes:
  • the embodiment of the present invention provides a decoding method, including:
  • the decoding matrix is a parity check PCM matrix, performing matrix transformation to separate the column HS corresponding to the readable data block;
  • the parity check PCM matrix includes a check block matrix and an identity matrix.
  • an embodiment of the present invention provides an apparatus for generating a decoding matrix, including:
  • the transformation module 100 is configured to perform matrix transformation on the constructed parity check PCM matrix to separate the unit failure matrix HL
  • the extraction module 200 is configured to remove the transformed PCM matrix from the unit failure matrix HL and obtain the column HS corresponding to the readable data block as a decoding matrix;
  • the parity check PCM matrix includes a check block matrix and a unit matrix
  • the unit failure matrix HL is a column corresponding to a failed data block in the transformed PCM matrix.
  • the transformation module 100 includes:
  • the separation unit is configured to perform matrix transformation on the PCM matrix so that the column corresponding to the failed data block is a unit matrix, and separates the columns corresponding to the failed data block to form a unit failure matrix HL.
  • the device described in this embodiment further includes: a marking module 300 configured to mark the column corresponding to the invalid data block in the PCM matrix.
  • the device described in this embodiment further includes:
  • the judging module 500 is configured to judge whether the column corresponding to the invalid data block is a unit matrix, and when it is not a unit matrix, the separation unit is enabled; when the unit matrix is current, the separation unit is not enabled.
  • An embodiment of the present invention provides a decoding device, including:
  • the decoding module is configured to obtain the original data block by multiplying the decoding matrix by the sum of the original data block and the check data block as 0;
  • the decoding matrix is a parity check PCM matrix, performing matrix transformation to separate the column HS corresponding to the readable data block;
  • the parity check PCM matrix includes a check block matrix and an identity matrix.
  • the embodiment of the present invention illustrates the process of generating a decoding matrix:

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Error Detection And Correction (AREA)
  • Detection And Correction Of Errors (AREA)

Abstract

一种生成解码矩阵的方法、解码方法和对应装置，涉及编码容错技术领域，所述方法包括：对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL（S101）；将变换后的PCM矩阵去除所述单位失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵（S102）；其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵，所述单位失效矩阵HL为变换后的PCM矩阵中失效数据块所对应的列。针对分布式系统环境，在不影响系统容错能力以及存储代价的前提下优化降级读写代价。

Description

一种生成解码矩阵的方法、解码方法和对应装置
本申请要求2019年05月28日递交的申请号为201910451394.5、发明名称为“一种生成解码矩阵的方法、解码方法和对应装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及编码容错技术领域，具体涉及一种生成解码矩阵的方法、解码方法和对应装置。
背景技术
目前，随着云计算和大数据技术的飞速发展，当前大数据中心也正在进行技术变革。随着数据中心的规模日益增大，硬件故障、软件失效的频率也在增加，这时存储系统的可靠性就显得尤为重要，数据的存储成本与可靠性都是分布式系统设计时需要考虑的因素。而纠删码可以在保证与其同样的数据可靠性的前提下，最小化系统的存储开销。目前业界分布式系统中普遍采用纠删码的方式进行存储，这种方式相较于多副本的存储方式可以在保证相同容错能力的条件下，节省接近一半的存储空间。但是，这种方式也存在着磁盘重构和降级读写时需要占用大量计算资源的问题；另一方面，不同于多副本的存储方式可以在一个磁盘失效时直接读取另一个磁盘上的副本，纠删码在一个数据磁盘失效后，需要读取额外的校验磁盘，并通过计算还原出原数据，这一过程被称为降级读写，计算的速度直接影响了系统读写的速度，同时影响了分布式系统的响应时间的问题。
传统的文件系统中的RaidFile中仅支持数据的Buffer模式的写入，当数据对于写入的延迟要求较高时，RaidFile并不能满足用户的写入需求。因此在磁盘失效时，为了应对用户的请求以及重新恢复数据，系统常常需要快速进行降级读写以及磁盘重构，这引起了计算资源的分配以及加速解码算法的问题。
发明内容
本申请提供一种生成解码矩阵的方法、解码方法和对应装置，针对分布式系统环境，在不影响系统容错能力以及存储代价的前提下优化降级读写代价。
采取的技术方案如下：
第一方面，本发明提供一种生成解码矩阵的方法，应用于分布式系统，包括：
对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL；
将变换后的PCM矩阵去除所述单位失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵，所述单位失效矩阵HL为变换后的PCM矩阵中失效数据块所对应的列。
优选地，对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL包括：
构造大小为m×n奇偶校验PCM矩阵，其中，n＝k+m，k为原始数据块的数量，m为校验数据块的数量，所述PCM矩阵的前k列为编码矩阵中生成的校验块矩阵，后m列为单位矩阵；
对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵，并将所述失效数据块所对应的列分离出来构成单位失效矩阵HL。
优选地，对所述PCM矩阵进行矩阵变换之前还包括：
标记所述PCM矩阵中失效数据块所对应的列。
优选地，对所述PCM矩阵进行矩阵变换之前还包括：
判断失效数据块所对应的列是否为单位矩阵，当不是单位矩阵时，执行矩阵变换的步骤；
当时单位矩阵时，跳过矩阵变换的步骤。
优选地，对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵包括：
对于失效数据块对应的列，记为{a1,a2,…,ar}；
对于任意i属于[1,r]，读取失效数据对应列ai；使其第i行的元素通过行变换变为1；
将ai列除第i行的元素外的剩余元素都通过行变换变为0。
优选地，使其第i行的元素通过行变换变为1包括：
将所述PCM矩阵的第i行所有元素都除以ai列的第i行的元素。
优选地，将ai列除第i行的元素外的剩余元素都通过行变换变为0包括：
对于PCM矩阵中的除第i行外的其他行，所述其他行在ai列中的元素x与第i行元素做乘法，再与这一行做减法操作。
第二方面，本发明提供一种解码方法，包括：
利用解码矩阵乘以原始数据块与校验数据块之和为0获得原始数据块；
所述解码矩阵为奇偶校验PCM矩阵进行矩阵变换分离出可读取的数据块所对应的列HS；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵
第三方面，本发明提供一种生成解码矩阵的装置，包括：
变换模块，设置为对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL；
抽出模块，设置为将变换后的PCM矩阵去除所述单位失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵，所述单位失效矩阵HL为变换后的PCM矩阵中失效数据块所对应的列。
对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL包括：
优选地，所述变换模块包括：
构造单元，设置为构造大小为m×n奇偶校验PCM矩阵，其中，n＝k+m，k为原始数据块的数量，m为校验数据块的数量，所述PCM矩阵的前k列为编码矩阵中生成的校验块矩阵，后m列为单位矩阵；
分离单元，设置为对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵，并将所述失效数据块所对应的列分离出来构成单位失效矩阵HL。
优选地，所述的装置还包括：标记模块，设置为标记所述PCM矩阵中失效数据块所对应的列。
优选地，所述的装置还包括：
判断模块，设置为判断所述失效数据块所对应的列是否为单位矩阵，当不是单位矩阵时，使能分离单元；当时单位矩阵时，不使能分离单元。
第四方面，本发明提供一种解码装置，包括：
解码模块，设置为利用解码矩阵乘以原始数据块与校验数据块之和为0获得原始数据块；
所述解码矩阵为奇偶校验PCM矩阵进行矩阵变换分离出可读取的数据块所对应的列HS；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵。
本申请和现有技术相比，具有如下有益效果：
本申请在分布式系统重构或降级读写的过程中，通过优化解码过程中的矩阵运算， 减少乘法运算次数，加速运算速度，从而减少分布式系统响应时间。
此外，还最大限度地提高编解码速度，以应对分布式系统不同场景下对响应时间的需求。
本申请通过优化解码矩阵加速解码时的矩阵运算的方法，即可以适用于任何一种通过矩阵运算进行编解码的纠删码，且不需要更改编码本身，有着广泛的适用性，并保证用户的请求响应的及时性和数据存储的可靠性。
附图说明
图1为本发明实施例的一种生成解码矩阵的方法的流程图；
图2为相关技术中的生成解码矩阵的示意图；
图3为本发明实施例的一种生成解码矩阵的装置的结构示意图；
图4为本发明实施例的一种生成解码矩阵的装置的结构示意图；
图5为本发明实施例的一种生成解码矩阵的装置的结构示意图；
图6为本发明实施例的分布式系统环境下的解码过程的流程图。
具体实施方式
下面将结合附图及实施例对本申请的技术方案进行更详细的说明。
需要说明的是，如果不冲突，本申请实施例以及实施例中的各个特征可以相互结合，均在本申请的保护范围之内。另外，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
在一种配置中，分布式系统包括多个数据分区，每个数据分区包括一个或者多个用于存储数据的数据盘，每个数据盘可包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存(memory)。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。内存可能包括一个或多个模块。
计算机可读介质包括永久性和非永久性、可移动和非可移动存储介质，可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、 只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)，快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。
实施例一
如图1所示，本发明实施例提供一种生成解码矩阵的方法，应用于分布式系统，包括：
S101、对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL；
S102、将变换后的PCM矩阵去除所述单位失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵，所述单位失效矩阵HL为变换后的PCM矩阵中失效数据块所对应的列。
本发明实施例中，步骤S101中，对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL包括：
构造大小为m×n奇偶校验PCM矩阵，其中，n＝k+m，k为原始数据块的数量，m为校验数据块的数量，所述PCM矩阵的前k列为编码矩阵中生成的校验块矩阵，后m列为单位矩阵；
对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵，并将所述失效数据块所对应的列分离出来构成单位失效矩阵HL。
PCM(Parity Check Matrix，奇偶校验矩阵)是一个描述所有磁盘节点中对应条带元素间线性关系的矩阵，是一个m×n(n＝k+m)的矩阵。前k列为编码矩阵中生成校验块的r×k的矩阵，后m列为一个r×r的单位矩阵。PCM矩阵与数据块和校验块组成的向量相乘的结果正好为0，因此描述了数据块与各个校验块间的线性关系，其中每一个矩阵的每一列都对应了一个相应的数据块。在相关技术中，通过将PCM矩阵中失效数据块所对应的列抽出，组成两个新的矩阵：HL代表失效数据块所对应的列组成的矩阵；HS代表可读取的数据块所对应的列，并将其分别放在等号的两边，分别乘以对应数据块，行成如图2所示的关系式。对关系式两边相乘HL的逆矩阵即可得到解码矩阵。但是在相关技术的算法中，HL矩阵的求逆并与HS矩阵做乘法是十分耗费计算资源的，因此本发明实施例从PCM矩阵生成解码矩阵的过程做了优化，首先对PCM矩阵进行矩阵变换，使分离出的HL为一个单位矩阵，由此可以减少HL的求逆运算和HL矩阵的逆矩阵与 HS矩阵的乘法运算。本发明实施例通过使用PCM矩阵来生成解码矩阵，并优化生成解码矩阵的运算过程，进一步减少耗费的计算资源以及计算时间，来保证数据块失效时，磁盘重构以及降级读写的速度，保证数据的可靠性以及系统的响应时间。
本发明实施例中，步骤S102对所述PCM矩阵进行矩阵变换之前还包括：标记所述PCM矩阵中失效数据块所对应的列；
本发明实施例中，步骤S102对所述PCM矩阵进行矩阵变换之前还包括：
判断失效数据块所对应的列是否为单位矩阵，当不是单位矩阵时，执行矩阵变换的步骤；
当时单位矩阵时，跳过矩阵变换的步骤。
本发明实施例中，步骤S102对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵包括：
对于失效数据块对应的列，记为{a1,a2,…,ar}；
对于任意i属于[1,r]，读取失效数据对应列ai；使其第i行的元素通过行变换变为1；
将ai列除第i行的元素外的剩余元素都通过行变换变为0。
本发明实施例对于失效数据块对应的PCM矩阵的列，分别对其进行标号，记为{a1,a2,…,ar}。对于任意i属于[1,r]，可以得到失效数据对应列ai。使其第i行的元素通过行变换变为1，之后，将ai列的剩余元素都通过行变换变为0。分别对i＝[1,r]进行上述操作，即可将HL矩阵变为一个单位矩阵。
对于PCM矩阵，由于等式右端为零矩阵，矩阵行变换不影响等式的成立。并且，由于原PCM矩阵中本身有一个单位矩阵存在，可以容易的通过行变换将HL矩阵变为单位矩阵，大幅减少了运算的步骤。
本发明实施例中，使其第i行的元素通过行变换变为1包括：
将所述PCM矩阵的第i行所有元素都除以ai列的第i行的元素。
本发明实施例中，将ai列除第i行的元素外的剩余元素都通过行变换变为0包括：
对于PCM矩阵中的除第i行外的其他行，所述其他行在ai列中的元素x与第i行元素做乘法，再与这一行做减法操作。
本发明实施例具有如下有益效果：
1、使用矩阵变换的方式减少了解码矩阵生成过程中，一个矩阵求逆和一个矩阵乘法的运算。
2、对所有使用矩阵进行编解码的纠删码都有加速运算的效果，适用性较广泛。
3、可以直接在解码矩阵生成中加入本方案，而不需要更改原纠删码方案。
实施例二
本实施例以奇偶校验PCM矩阵的大小为m×n例说明生成解码矩阵的过程：
构造大小为m×n奇偶校验PCM矩阵，其中，n＝k+m，k为原始数据块的数量，m为校验数据块的数量，所述PCM矩阵的前k列为编码矩阵中生成的校验块矩阵，后m列为单位矩阵；
对所述PCM矩阵进行矩阵变换，使得分离出的失效矩阵HL为单位矩阵，所述失效矩阵HL为矩阵变换后抽出的失效数据块所对应的列；
将变换后的PCM矩阵去除失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵。
本实施例，对所述PCM矩阵进行矩阵变换之前还包括：
标记所述PCM矩阵中失效数据块所对应的列。
本实施例，对所述PCM矩阵进行矩阵变换之前还包括：
判断所述失效矩阵HL是否为单位矩阵，当不是单位矩阵时，执行矩阵变换的步骤；
当时单位矩阵时，跳过矩阵变换的步骤。
实施例三
本发明实施例提供一种解码方法，包括：
利用解码矩阵乘以原始数据块与校验数据块之和为0获得原始数据块；
所述解码矩阵为奇偶校验PCM矩阵进行矩阵变换分离出可读取的数据块所对应的列HS；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵。
实施例四
如图3所示，本发明实施例提供一种生成解码矩阵的装置，包括：
变换模块100，设置为对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL
抽出模块200，设置为将变换后的PCM矩阵去除所述单位失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵，所述单位失效矩阵HL为变换后的PCM矩阵中失效数据块所对应的列。
对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL包括：
如图4所示，所述变换模块100包括：
构造单元，设置为构造大小为m×n奇偶校验PCM矩阵，其中，n＝k+m，k为原始数据块的数量，m为校验数据块的数量，所述PCM矩阵的前k列为编码矩阵中生成的校验块矩阵，后m列为单位矩阵；
分离单元，设置为对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵，并将所述失效数据块所对应的列分离出来构成单位失效矩阵HL。
如图4所示，本实施例所述的装置，还包括：标记模块300，设置为标记所述PCM矩阵中失效数据块所对应的列。
如图5所示，本实施例所述的装置，还包括：
判断模块500，设置为判断所述失效数据块所对应的列是否为单位矩阵，当不是单位矩阵时，使能分离单元；当时单位矩阵时，不使能分离单元。
实施例五
本发明实施例提供一种解码装置，包括：
解码模块，设置为利用解码矩阵乘以原始数据块与校验数据块之和为0获得原始数据块；
所述解码矩阵为奇偶校验PCM矩阵进行矩阵变换分离出可读取的数据块所对应的列HS；
其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵。
实施例六
如图6所示，本发明实施例说明生成解码矩阵的过程：
生成一个m×n(n＝k+m)的PCM矩阵，前k列为编码矩阵中生成校验块的r×k的矩阵，后m列为一个r×r的单位矩阵；
标记所述PCM矩阵中失效数据块所对应的列{a1,a2,…,ar}，并使i＝0；
判断{a1,a2,…,ar}是否可以组成一个单位矩阵；
如果是，则从所述PCM矩阵中去除失效数据块所对应的列{a1,a2,…,ar}得到解码矩阵；
如果否，则令PCM矩阵的第i行所有元素都除以该第ai列的元素，使该元素变为1；将第ai列中，除第i行元素外的其他元素，都通过与这一行做减法操作变为0；从所述PCM矩阵中去除失效数据块所对应的列{a1,a2,…,ar}得到解码矩阵。
虽然本发明所揭示的实施方式如上，但其内容只是为了便于理解本发明的技术方案 而采用的实施方式，并非用于限定本发明。任何本发明所属技术领域内的技术人员，在不脱离本发明所揭示的核心技术方案的前提下，可以在实施的形式和细节上做任何修改与变化，但本发明所限定的保护范围，仍须以所附的权利要求书限定的范围为准。

Claims (13)

  1. 一种生成解码矩阵的方法，应用于分布式系统，其特征在于，包括：
    对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL；
    将变换后的PCM矩阵去除所述单位失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵；
    其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵，所述单位失效矩阵HL为变换后的PCM矩阵中失效数据块所对应的列。
  2. 如权利要求1所述的方法，其特征在于，对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL包括：
    构造大小为m×n奇偶校验PCM矩阵，其中，n＝k+m，k为原始数据块的数量，m为校验数据块的数量，所述PCM矩阵的前k列为编码矩阵中生成的校验块矩阵，后m列为单位矩阵；
    对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵，并将所述失效数据块所对应的列分离出来构成单位失效矩阵HL。
  3. 如权利要求2所述的方法，其特征在于，对所述PCM矩阵进行矩阵变换之前还包括：
    标记所述PCM矩阵中失效数据块所对应的列。
  4. 如权利要求2所述的方法，其特征在于，对所述PCM矩阵进行矩阵变换之前还包括：
    判断失效数据块所对应的列是否为单位矩阵，当不是单位矩阵时，执行矩阵变换的步骤；
    当时单位矩阵时，跳过矩阵变换的步骤。
  5. 如权利要求2所述的方法，其特征在于，对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵包括：
    对于失效数据块对应的列，记为{a1,a2,…,ar}；
    对于任意i属于[1,r]，读取失效数据对应列ai；使其第i行的元素通过行变换变为1；
    将ai列除第i行的元素外的剩余元素都通过行变换变为0。
  6. 如权利要求5所述的方法，其特征在于，使其第i行的元素通过行变换变为1包括：
    将所述PCM矩阵的第i行所有元素都除以ai列的第i行的元素。
  7. 如权利要求5所述的方法，其特征在于，将ai列除第i行的元素外的剩余元素都通过行变换变为0包括：
    对于PCM矩阵中的除第i行外的其他行，所述其他行在ai列中的元素x与第i行元素做乘法，再与这一行做减法操作。
  8. 一种解码方法，其特征在于，包括：
    利用解码矩阵乘以原始数据块与校验数据块之和为0获得原始数据块；
    所述解码矩阵为奇偶校验PCM矩阵进行矩阵变换分离出可读取的数据块所对应的列HS；
    其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵。
  9. 一种生成解码矩阵的装置，其特征在于，包括：
    变换模块，设置为对构造的奇偶校验PCM矩阵进行矩阵变换，分离出单位失效矩阵HL；
    抽出模块，设置为将变换后的PCM矩阵去除所述单位失效矩阵HL得到的可读取的数据块所对应的列HS作为解码矩阵；
    其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵，所述单位失效矩阵HL为变换后的PCM矩阵中失效数据块所对应的列。
  10. 如权利要求9所述的装置，其特征在于，所述变换模块包括：
    构造单元，设置为构造大小为m×n奇偶校验PCM矩阵，其中，n＝k+m，k为原始数据块的数量，m为校验数据块的数量，所述PCM矩阵的前k列为编码矩阵中生成的校验块矩阵，后m列为单位矩阵；
    分离单元，设置为对所述PCM矩阵进行矩阵变换，使得失效数据块所对应的列为单位矩阵，并将所述失效数据块所对应的列分离出来构成单位失效矩阵HL。
  11. 如权利要求10所述的装置，其特征在于，还包括：标记模块，设置为标记所述PCM矩阵中失效数据块所对应的列。
  12. 如权利要求10所述的装置，其特征在于，还包括：
    判断模块，设置为判断所述失效数据块所对应的列是否为单位矩阵，当不是单位矩阵时，使能分离单元；当时单位矩阵时，不使能分离单元。
  13. 一种解码装置，其特征在于，包括：
    解码模块，设置为利用解码矩阵乘以原始数据块与校验数据块之和为0获得原始数据块；
    所述解码矩阵为奇偶校验PCM矩阵进行矩阵变换分离出可读取的数据块所对应的列HS；
    其中，所述奇偶校验PCM矩阵包括校验块矩阵和单位矩阵。
PCT/CN2020/091466 2019-05-28 2020-05-21 一种生成解码矩阵的方法、解码方法和对应装置 WO2020238736A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910451394.5 2019-05-28
CN201910451394.5A CN112015325B (zh) 2019-05-28 2019-05-28 一种生成解码矩阵的方法、解码方法和对应装置

Publications (1)

Publication Number Publication Date
WO2020238736A1 true WO2020238736A1 (zh) 2020-12-03

Family

ID=73500611

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/091466 WO2020238736A1 (zh) 2019-05-28 2020-05-21 一种生成解码矩阵的方法、解码方法和对应装置

Country Status (2)

Country Link
CN (1) CN112015325B (zh)
WO (1) WO2020238736A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114244374B (zh) * 2021-11-22 2022-08-05 成都博尔微晶科技有限公司 一种校验矩阵生成方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090292966A1 (en) * 2008-05-23 2009-11-26 Deutsches Zentrum Fuer Luft-Und Raumfahrt E.V. Method for recovery of lost and/or corrupted data
US20130173956A1 (en) * 2011-12-30 2013-07-04 Streamscale, Inc. Using parity data for concurrent data authentication, correction, compression, and encryption
CN103678029A (zh) * 2013-12-18 2014-03-26 华中科技大学 一种容多个设备和扇区错的编码的优化方法
CN104850468A (zh) * 2015-05-31 2015-08-19 上海交通大学 基于校验矩阵的纠删码解码方法
CN108132854A (zh) * 2018-01-15 2018-06-08 成都信息工程大学 一种可同时恢复数据元素及冗余元素的纠删码解码方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4545793B2 (ja) * 2004-08-10 2010-09-15 サムスン エレクトロニクス カンパニー リミテッド ブロック低密度パリティ検査符号を符号化／復号化する装置及び方法
WO2014069464A1 (ja) * 2012-11-05 2014-05-08 三菱電機株式会社 誤り訂正符号化方法および誤り訂正符号化装置
US9184767B2 (en) * 2013-09-19 2015-11-10 SanDisk Technologies, Inc. Scoring variable nodes for low density parity check code decoding

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090292966A1 (en) * 2008-05-23 2009-11-26 Deutsches Zentrum Fuer Luft-Und Raumfahrt E.V. Method for recovery of lost and/or corrupted data
US20130173956A1 (en) * 2011-12-30 2013-07-04 Streamscale, Inc. Using parity data for concurrent data authentication, correction, compression, and encryption
CN103678029A (zh) * 2013-12-18 2014-03-26 华中科技大学 一种容多个设备和扇区错的编码的优化方法
CN104850468A (zh) * 2015-05-31 2015-08-19 上海交通大学 基于校验矩阵的纠删码解码方法
CN108132854A (zh) * 2018-01-15 2018-06-08 成都信息工程大学 一种可同时恢复数据元素及冗余元素的纠删码解码方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JUNQING GU ET AL.: "Optimizing the Parity-Check Matrix for Efficient Decoding of RS-based Cloud Storage Systems", 2019 IEEE INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS), 2 September 2019 (2019-09-02), XP033610336, DOI: 20200807151913 *

Also Published As

Publication number Publication date
CN112015325B (zh) 2024-03-26
CN112015325A (zh) 2020-12-01

Similar Documents

Publication Publication Date Title
WO2018000812A1 (zh) 数据存储方法及装置
US8352490B2 (en) Method and system for locating update operations in a virtual machine disk image
US10540119B2 (en) Distributed shared log storage system having an adapter for heterogenous big data workloads
CN112579602B (zh) 多版本数据存储方法、装置、计算机设备及存储介质
US10387307B2 (en) Lock-free raid implementation in multi-queue architecture
TW201220197A (en) for improving the safety and reliability of data storage in a virtual machine based on cloud calculation and distributed storage environment
CN110750382A (zh) 用于提高数据修复性能的最小存储再生码编码方法及系统
WO2023116238A1 (zh) 基于纠删码的编码方法、分布式系统、设备及存储介质
CN111708738A (zh) 实现hadoop文件系统hdfs与对象存储s3数据互访方法及系统
WO2023056904A1 (zh) 校验块的生成方法及装置
WO2020238736A1 (zh) 一种生成解码矩阵的方法、解码方法和对应装置
WO2019109256A1 (zh) 一种日志管理方法、服务器和数据库系统
CN115098046A (zh) 磁盘阵列初始化方法、系统、电子设备及存储介质
WO2021151298A1 (zh) 一种数据冗余处理方法、装置、设备及存储介质
AU2014301874B2 (en) Data writing method and memory system
WO2021012164A1 (zh) 数据重构的方法、装置、计算机设备、存储介质及系统
CN112435157A (zh) 包括不同类型的存储器装置的图形处理系统及其操作方法
US20240160528A1 (en) Data Storage Method and Apparatus in Storage System
EP4170499A1 (en) Data storage method, storage system, storage device, and storage medium
CN113421095A (zh) 一种区块链交易并行执行加速方法
US20230161754A1 (en) Data processing method and apparatus, electronic device, and storage medium
WO2020238653A1 (zh) 一种分布式系统环境下的编码方法、解码方法和对应装置
JP2016537708A (ja) メモリへの同時アクセス
CN115629901A (zh) 日志回放方法及装置、数据恢复方法及装置、电子设备
JP2008097156A (ja) 記憶制御装置、記憶制御方法および記憶制御プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20815203

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20815203

Country of ref document: EP

Kind code of ref document: A1