CN114253684A - Erasure task processing system, method, electronic device and storage medium - Google Patents

Erasure task processing system, method, electronic device and storage medium Download PDF

Info

Publication number
CN114253684A
CN114253684A CN202111444997.6A CN202111444997A CN114253684A CN 114253684 A CN114253684 A CN 114253684A CN 202111444997 A CN202111444997 A CN 202111444997A CN 114253684 A CN114253684 A CN 114253684A
Authority
CN
China
Prior art keywords
data
slice
erasure
task
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111444997.6A
Other languages
Chinese (zh)
Inventor
王明明
张磊
吴睿振
王凛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Original Assignee
Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd filed Critical Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Priority to CN202111444997.6A priority Critical patent/CN114253684A/en
Publication of CN114253684A publication Critical patent/CN114253684A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
    • G06F12/0815—Cache consistency protocols
    • G06F12/0831—Cache consistency protocols using a bus scheme, e.g. with bus monitoring or watching means
    • G06F12/0835—Cache consistency protocols using a bus scheme, e.g. with bus monitoring or watching means for main memory peripheral accesses (e.g. I/O or DMA)
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0613—Improving I/O performance in relation to throughput
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Detection And Correction Of Errors (AREA)

Abstract

The application discloses erasure correcting task processing system, erasure correcting task processing system includes: the data slice scheduling unit is used for determining slice parameters corresponding to the erasure correcting tasks and sending the slice parameters to the input data address calculation unit and the output data address calculation unit; an input data address calculation unit for calculating first address information of each input slice data block according to the slice parameters; an output data address calculation unit for calculating second address information of each output slice data block according to the slice parameters; and the DMA unit is used for controlling the input and the output of the erasure correcting processing module according to the first address information and the second address information so as to complete the erasure correcting task. The method and the device can reduce the occupation of erasure task processing on computing resources. The application also discloses an erasure task processing method, a storage medium and an electronic device, which have the beneficial effects.

Description

Erasure task processing system, method, electronic device and storage medium
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to an erasure task processing system, an erasure task processing method, an erasure task processing electronic device, and a storage medium.
Background
In the face of the storage requirement of mass data, the distributed storage gradually replaces the dominant position of unified storage by the advantages of low cost, good expandability and the like, and has gained more and more attention in the aspects of theoretical research and practical application. On the other hand, a distributed storage system usually comprises a plurality of nodes, and the system often has node failure due to software and hardware failures, human errors and the like. In order to improve the data reliability of the distributed storage system and ensure that the data collection node can realize the reconstruction of the original file with high probability, a certain amount of redundancy needs to be additionally stored on the basis of storing the original data, so that the system can still normally operate under the condition that partial nodes fail, and the data collection node can still realize decoding recovery of the original file. Meanwhile, in order to maintain the reliability of the system, the failed node needs to be repaired in time, so that it is very important to design a good node repair mechanism.
In distributed storage, the solution of open source software such as intel-ISA, jerasury 2.0 and the like is mostly used in the industry for encoding and decoding based on erasure codes, that is, RS (Reed-Solomon Code) related software is run on a CPU for erasure encoding and decoding. However, the conventional erasure task processing method needs to occupy more computing resources.
Therefore, how to reduce the occupation of the computation resources by the erasure task processing is a technical problem that needs to be solved by those skilled in the art at present.
Disclosure of Invention
The application aims to provide an erasure correction task processing system, an erasure correction task processing method, an electronic device and a storage medium, which can reduce occupation of CPU computing resources by erasure correction task processing.
In order to solve the above technical problem, the present application provides an erasure task processing system, where the state early warning system includes:
the data slice scheduling unit is used for determining slice parameters corresponding to the erasure correcting tasks and sending the slice parameters to the input data address calculation unit and the output data address calculation unit; wherein the slice parameters include: presetting a slice size, a data block size of the erasure correcting task, the number of input data blocks and the number of output data blocks;
the input data address calculation unit is used for calculating first address information of each input slice data block according to the slice parameters and sending the first address information to the DMA unit; wherein the first address information comprises a start address and a data size of the input slice data block;
the output data address calculation unit is used for calculating second address information of each output slice data block according to the slice parameters and sending the second address information to the DMA unit; wherein the second address information comprises a start address and a data size of the output slice data block;
and the DMA unit is used for controlling the input and the output of the erasure correcting processing module according to the first address information and the second address information so as to complete the erasure correcting task.
Optionally, the data slice scheduling unit includes:
the slice parameter determining subunit is used for setting the size of the preset slice according to the scheduling mode of the DMA unit and the caching capacity output by the hardware;
or, the parameter adaptive adjustment subunit is configured to adaptively adjust the size of the preset slice according to the current buffer idle condition.
Optionally, the erasure correcting processing module includes a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, an output data scheduling unit, a processing unit, and an output data scheduling unit;
the task scheduling unit is used for receiving and analyzing the erasure correction task to obtain analysis information, and sending the analysis information to the data scheduling unit, the matrix scheduling unit, the output data scheduling unit and the data slice scheduling unit;
optionally, the data scheduling unit is configured to input each input slice data block into the processing unit according to the number of input data blocks, the number of output data blocks, and the size of each input slice data block.
Optionally, the matrix scheduling unit is configured to determine an encoding matrix and/or a decoding matrix according to the number of the input data blocks and the number of the output data blocks, and input the determined encoding matrix and/or decoding matrix to the processing unit.
Optionally, the configuration scheduling unit is configured to generate configuration information according to the number of the input data blocks, the number of the output data blocks, and the size of each input slice data block, and input the configuration information into the processing unit and the output data scheduling unit.
Optionally, the output data scheduling unit is configured to determine whether the erasure correcting task is completed according to the configuration information; and the task scheduling unit is also used for returning response information to the task scheduling unit after the erasure task is completed.
The application also provides an erasure task processing method, which comprises the following steps:
determining a slice parameter corresponding to the erasure correcting task; wherein the slice parameters include: presetting a slice size, a data block size of the erasure correcting task, the number of input data blocks and the number of output data blocks;
calculating first address information of each input slice data block according to the slice parameters; wherein the first address information comprises a start address and a data size of the input slice data block;
calculating second address information of each output slice data block according to the slice parameters; wherein the second address information comprises a start address and a data size of the output slice data block;
and controlling the input and the output of an erasure processing module according to the first address information and the second address information so as to complete the erasure task.
The application also provides a storage medium, on which a computer program is stored, which when executed implements the steps performed by the above erasure task processing method.
The application also provides an electronic device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the execution of the erasure correcting task processing method when calling the computer program in the memory.
The application provides an erasure task processing system, which comprises: the data slice scheduling unit is used for determining slice parameters corresponding to the erasure correcting tasks and sending the slice parameters to the input data address calculation unit and the output data address calculation unit; wherein the slice parameters include: presetting a slice size, a data block size of the erasure correcting task, the number of input data blocks and the number of output data blocks; the input data address calculation unit is used for calculating first address information of each input slice data block according to the slice parameters and sending the first address information to the DMA unit; wherein the first address information comprises a start address and a data size of the input slice data block; the output data address calculation unit is used for calculating second address information of each output slice data block according to the slice parameters and sending the second address information to the DMA unit; wherein the second address information comprises a start address and a data size of the output slice data block; and the DMA unit is used for controlling the input and the output of the erasure correcting processing module according to the first address information and the second address information so as to complete the erasure correcting task.
The data slice scheduling unit is used for determining slice parameters of erasure tasks, and the input data address calculation unit can determine the initial address and the data size of each input slice data block according to the slice parameters, so that the DMA module controls the input of the erasure processing module according to the initial address and the data size of each input slice data block. The output data address calculation unit may determine a start address and a data size of each output slice data block according to the slice parameter, so that the DMA module controls an output of the erasure processing module according to the start address and the data size of each output slice data block. According to the scheme, data slicing processing is carried out on the data blocks of the erasure correcting task, and erasure correction of large data blocks is achieved by dynamically rolling and scheduling small data blocks. The process has no perception to upper application software, and can be suitable for hardware systems cached on different chips in a configurable slicing size and hardware slicing mode. The method and the device can reduce occupation of the erasure task processing on the CPU computing resources. The application also provides an erasure task processing method, a storage medium and an electronic device, which have the beneficial effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings needed for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a schematic diagram of an erasure correction application in a distributed storage node according to an embodiment of the present application;
fig. 2 is a block diagram of RS erasure coding and decoding hardware provided in an embodiment of the present application;
fig. 3 is a flowchart illustrating a working procedure of RS erasure coding and decoding hardware according to an embodiment of the present disclosure;
fig. 4 is a schematic diagram of two consecutive task schedules according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
An erasure task processing system provided by an embodiment of the present application includes:
the data slice scheduling unit is used for determining slice parameters corresponding to the erasure correcting tasks and sending the slice parameters to the input data address calculation unit and the output data address calculation unit; wherein the slice parameters include: presetting a slice size, a data block size of the erasure correcting task, the number of input data blocks and the number of output data blocks;
the input data address calculation unit is used for calculating first address information of each input slice data block according to the slice parameters and sending the first address information to the DMA unit; wherein the first address information comprises a start address and a data size of the input slice data block;
the output data address calculation unit is used for calculating second address information of each output slice data block according to the slice parameters and sending the second address information to the DMA unit; wherein the second address information comprises a start address and a data size of the output slice data block;
and the DMA unit is used for controlling the input and the output of the erasure correcting processing module according to the first address information and the second address information so as to complete the erasure correcting task.
After receiving the erasure correction task, the embodiment determines the data block that needs to be input for executing the erasure correction task, and divides the data block into a plurality of input slice data blocks, so as to perform erasure correction processing on the plurality of input slice data blocks. The present embodiment calculates a start address and a data size of each input slice data using an input data address calculation unit. The present embodiment also calculates the start address and data size of each output slice data block using the output data address calculation unit. The present embodiment transmits the start address and the data size of the input slice data block and the start address and the data size of the output slice data block to a DMA (Direct Memory Access) unit. The DMA unit may control the erasure correction processing module to perform data input according to the start address and data size of the input slice data block, and may also control the erasure correction processing module to perform data output according to the start address and data size of the output slice data block.
In this embodiment, the data slice scheduling unit is used to determine the slice parameters of the erasure correction task, and the input data address calculation unit may determine the start address and the data size of each input slice data block according to the slice parameters, so that the DMA module controls the input of the erasure correction processing module according to the start address and the data size of each input slice data block. The output data address calculation unit may determine a start address and a data size of each output slice data block according to the slice parameter, so that the DMA module controls an output of the erasure processing module according to the start address and the data size of each output slice data block. According to the scheme, data slicing processing is carried out on the data blocks of the erasure correcting task, and erasure correction of large data blocks is achieved by dynamically rolling and scheduling small data blocks. The process has no perception to upper application software, and can be suitable for hardware systems cached on different chips in a configurable slicing size and hardware slicing mode.
Further, the data slice scheduling unit includes:
the slice parameter determining subunit is used for setting the size of the preset slice according to the scheduling mode of the DMA unit and the caching capacity output by the hardware;
or, the parameter adaptive adjustment subunit is configured to adaptively adjust the size of the preset slice according to the current buffer idle condition.
The data slice scheduling unit may use the slice parameter determining subunit to perform static configuration on the preset slice size, or may use the parameter adaptive adjusting subunit to perform dynamic configuration on the preset slice size. The buffering capacity of the hardware output comprises a buffering size and a throughput rate, and the preset slice size is the size of an input slice data block and an output slice data block. In this embodiment, a self-adaptive method is introduced to slice data, the sizes of several caches can be divided on the basis of a certain determined cache size, scheduling of different slice sizes is realized according to a real-time cache idle condition, and the utilization rate of the cache and the throughput rate of the system are further improved.
Further, the erasure correcting processing module includes a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, an output data scheduling unit, a processing unit, and an output data scheduling unit.
And the task scheduling unit is used for receiving and analyzing the erasure correction task to obtain analysis information, and sending the analysis information to the data scheduling unit, the matrix scheduling unit, the output data scheduling unit and the data slice scheduling unit.
And the data scheduling unit is used for inputting each input slice data block into the processing unit according to the number of the input data blocks, the number of the output data blocks and the size of each input slice data block.
And the matrix scheduling unit is used for determining an encoding matrix and/or a decoding matrix according to the number of the input data blocks and the number of the output data blocks and inputting the determined encoding matrix and/or decoding matrix into the processing unit.
And the configuration scheduling unit is used for generating configuration information according to the number of the input data blocks, the number of the output data blocks and the size of each input slice data block, and inputting the configuration information into the processing unit and the output data scheduling unit.
The output data scheduling unit is used for judging whether the erasure correcting task is completed or not according to the configuration information; and the task scheduling unit is also used for returning response information to the task scheduling unit after the erasure task is completed.
The flow described in the above embodiment is explained below by an embodiment in practical use.
Erasure Code (Erasure Code) belongs to a forward error correction technique in the coding theory, and is applied to the communication field for the first time to solve the problems of loss and loss in data transmission. Erasure coding techniques have been introduced into the storage area because of their superior effectiveness in preventing data loss. Erasure codes can effectively reduce storage overhead while ensuring the same reliability, and therefore erasure code technology is widely applied to various large storage systems and data centers, such as, for example, Azure by microsoft, F4 by Facebook, and the like.
The erasure codes are of various types, and RS codes (Reed-Solomon codes) applied in a distributed environment are more common in a real storage system. The RS code is associated with two parameters k and r. Given two positive integers k and r, the RS code encodes k data blocks into r additional check blocks. The way in which r parity chunks are encoded based on a vandermonde matrix or a cauchy matrix is referred to as RS erasure coding using vandermonde matrix or cauchy matrix encoding.
RS erasure codes based on vandermonde matrices are as follows:
Figure BDA0003383832830000071
the RS erasure code based on the Cauchy matrix is as follows:
Figure BDA0003383832830000072
the upper k matrix corresponds to k original data blocks, and the lower r matrix corresponds to a coding matrix, which is obtained by correlating the coding matrix with original data D1To DkMultiplying to obtain newly added P1To PrThe resulting r check data are encoded. When any data less than r is in error or lost in transmission and needs to be corrected, the inverse matrix of the matrix corresponding to the residual data is used for multiplying the data (the multiplication is the multiplication operation in GF field), namely, the original data block D is obtained1To Dk. Gf (galois field) is the galois field.
For decoding with D1 to Dr data loss as an example, the RS erasure code recovers the data as follows:
Figure BDA0003383832830000081
the core concept of erasure codes is to construct a reversible coding matrix to generate the parity data, and the inverse matrix can be calculated to recover the original data. Common RS erasure codes use the above-described cauchy matrix or vandermonde matrix, which has the advantage that the resulting matrix is definitely reversible, any sub-matrix thereof is also reversible, and the size expansion of the matrix is simple.
D in the above introduction1To Dk，P1To PrMay be 8 bits, 16 bits or other values (referred to herein as sign bits, referred to as symbols) depending on the bit width of the multiplication operation, i.e., the finite field definition of a galois field operation. For a task, if the block size is Z bits. For multiplication, if 8-bit Galois field operation is used, it is split into (Z/8) independent data and the same matrix for operation. And outputs the operation result. The 1 st byte of code output can be found as follows, and an erasure code with a block size of 4Kbyte can be found by repeating 4096 times of different data and same matrix calculations. Therefore, the erasure correction is characterized by each data block and check block, and different symbols are irrelevant.
The erasure coding method for data with a block size of 4Kbyte is as follows:
Figure BDA0003383832830000082
calculating the 1 st byte;
as can be seen from the above analysis, for one erasure correction hardware system, to complete erasure correction coding or decoding function, the following three elements need to be provided:
and (3) erasure correction task allocation: number of data blocks, number of check blocks, task type (encoding or decoding), status of data blocks (present, erroneous or missing), status of check blocks (present, erroneous or missing).
Matrix: for encoding, the original encoding matrix needs to be provided, for decoding, the decoding matrix needs to be provided, or the original encoding matrix is provided, and the decoding is completed by hardware through Gaussian elimination calculation of the inverse matrix.
Inputting data: for encoding, data of all data blocks needs to be provided; for decoding, data of the existence data block and data of the existence check block need to be provided.
In addition, regarding erasure performance, that is, throughput of data, is also a factor to be considered, and it can be seen from the above description that the throughput of data can be improved only by performing as much parallelism as possible for each symbol. In summary, to achieve a high performance erasure correction system, a highly concurrent hardware system is required. It is now more used in the industry to run open source software, such as ISA of Intel, jerasure2.0 open source software, etc., on a general-purpose CPU, either to increase the throughput of Data through a higher operating frequency of the CPU, or to accelerate the Data through a hardware acceleration Instruction provided by the CPU, such as SIMD (Single Instruction Multiple Data) of Intel.
In distributed storage in the industry, the solutions of open source software such as intel-ISA and jerasure2.0 are mostly used for encoding and decoding based on RS erasure codes, that is, RS-related software is run on a CPU for erasure encoding and decoding. One obvious problem with this solution is that it occupies the computational resources of the CPU. The method and the device adopt a scheme of a hardware acceleration scheme to carry out erasure coding, decoding can reduce the cost of a CPU, and meanwhile, the throughput rate is higher than that of the CPU scheme.
Referring to fig. 1, fig. 1 is a schematic diagram of an erasure correction application in a distributed storage node according to an embodiment of the present application, where a distributed storage system includes the distributed storage node, an erasure correction hardware processing unit, and storage pools 1 to 3, and after receiving a task request, the erasure correction hardware processing unit may perform a task response. In fig. 1, D denotes a data block and C denotes a parity block. In a distributed storage system, a processing node may process many erasure policies, for example, a storage node may have different policy configurations such as 4+2, 6+2, 3+1, etc. In the case of multiprocessing or multithreading concurrency, the tasks that need to be processed are also different for a hardware system that handles erasure. As shown below, on a distributed storage node, there are multiple storage pools. For example, the erasure-checking policy for the three storage pools in the illustration is 4+2, 6+2, or 2+ 1. The host storage node initiates 5 continuous tasks to the hardware processing unit, and the hardware processing unit needs to process the erasure correcting tasks in sequence and feed back the processed data after encoding/decoding to the host.
The present embodiment provides a hardware implementation of a data slicing technique for data scheduling in an erasure correcting hardware system, where the hardware may automatically calculate the size of a data block of each data scheduling according to the size of a data block of a target task defined by a user, and may use a relatively small on-chip cache to achieve a very high data throughput. The hardware can automatically calculate the size of an input data block and the source address of input data of each scheduling, and can automatically calculate the size of an output data block and the destination address of output data of each scheduling.
The environment available for this embodiment is as follows: for k data blocks and r check blocks, the coding needs to configure the number k of the data blocks, the number r of the check blocks, a coding matrix, a decoding matrix and block size information. The decoding needs to configure the Number k of data blocks and the Number r of check blocks, a coding matrix or a decoding matrix, the Block size, the missing data blocks, the existing check Block information, and the sum of the missing data blocks and the check blocks is lbn (Lost Block Number), and as long as lbn is less than or equal to r, erasure correcting decoding can recover all the missing data blocks and check blocks. The data slice size is realized by means of static configuration, and the slice size is determined by the caching capacity of hardware output in combination with a DMA scheduling mode of a system. The slice size is not associated with the size of the data block and may be greater than, less than, or equal to the size of the data block. In this embodiment, operations such as data scheduling and matrix scheduling may be performed by externally assisted hardware, such as DMA.
Referring to fig. 2, fig. 2 is a block diagram of RS erasure correcting coding and decoding hardware provided in the embodiment of the present application, and as shown in fig. 2, the RS erasure correcting hardware provided in the embodiment uses an unlimited maximum number of supported data nodes, and the maximum number r of supported check nodes is 6 as an example.
The module is divided into a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, a configuration scheduling unit, Galois multiplication units (6), an output data scheduling unit (6), a data slice scheduling unit, an input data address calculation unit, an output data address calculation unit and a Galois multiplication unit, namely a processing unit. The functions of the specific modules are described as follows:
the task scheduling unit is responsible for receiving and analyzing the erasure task request and sending data information needing scheduling and inputting to the input data scheduling unit according to the information obtained by analysis; sending matrix information required by encoding and decoding to a matrix scheduling unit; and sending the specific configuration information of the task to a configuration scheduling unit. And after the erasure task processing is finished, the erasure task processing module is responsible for sending a response of task completion to the outside.
Secondly, the input data scheduling unit sends the input data of the erasure correction task to a PE (Process element) processing unit needing to be calculated according to the input data needing to be scheduled, and the scheduling unit also needs to ensure that data loss does not occur, namely data flow control is needed to be necessary for a target PE.
And thirdly, the matrix scheduling unit is responsible for scheduling a matrix required by erasure-correcting coding or decoding from the outside according to the erasure-correcting task, or the matrix scheduling unit calculates a required decoding matrix according to the original coding matrix. And meanwhile, the encoding or decoding matrix is sent to a PE target unit needing to be calculated, and the scheduling unit also needs to ensure that the loss of the matrix, namely the data flow control necessary for the target PE does not occur.
And fourthly, the configuration scheduling unit issues the configuration information to the target PE and the output data scheduling unit according to the erasure task, and the scheduling unit also needs to ensure that the loss of the configuration information does not occur, namely the necessary data flow control with the target PE and the output data scheduling unit is required.
And the Galois multiplication unit calculates the corresponding matrix according to the sent configuration, data and matrix.
And sixthly, the output data scheduling unit outputs the generated data according to the sent configuration, counts the output data of the target PE, finishes the processing of the task when the output data number is consistent with the expected data number of the task, and sends a notice of finishing the task to the task scheduling unit.
And the data slice scheduling unit is responsible for distributing specific data switching tasks to the input data scheduling unit and the output data scheduling unit according to the types of the tasks.
And the input data address calculation unit is responsible for calculating the start address and the data block size of each scheduled slice data block according to the start address, the slice size and the data block size of each input data block, and scheduling the input data from the outside.
And ninthly, the output data address calculation unit is responsible for calculating the start address and the data block size of each scheduled slice data block according to the start address, the slice size and the data block size of each output data block and scheduling and outputting the generated output data to the outside.
Referring to fig. 3, fig. 3 is a flowchart illustrating a working process of RS erasure correcting coding/decoding hardware according to an embodiment of the present application, where the specific process is as follows:
the task scheduling unit obtains an erasure task request.
The task scheduler pushes the number K of input data blocks required for the erasure correction task, the number lbn of output data blocks, and the size of each data block into the input data scheduler. The input data scheduling unit pushes lbn target PEs (i.e., galois multiplication units) in a predetermined data order based on the number K of input data blocks, the number lbn of output data blocks, and the size of each data block, and the data received by the target PEs are identical.
The task scheduling unit pushes the number K of input data blocks and the number lbn of output data blocks required by the erasure correcting task to the matrix scheduling unit. The matrix scheduler unit sends a matrix of lbn × K size to lbn target PEs (each PE receives K matrix elements) based on the number of input data blocks K, the number of output data blocks lbn.
The task scheduler pushes the number K of input data blocks, the number lbn of output data blocks, and the size of each data block, which are required for the erasure correction task, to the placement scheduler. The configuration scheduling unit sends configuration information to lbn target PEs (each PE receives the same configuration information) according to the number of output data blocks lbn. The configuration scheduling unit sends the configuration information to lbn output data scheduling units (each unit receives the same configuration information) according to the number of output data blocks lbn.
The data slice scheduling unit notifies two address calculation units (i.e., an input data address calculation unit and an output data address calculation unit) according to the configured slice size and the data block size of the entire task, as well as the number of output data blocks K and the number of output data blocks lbm.
The input data address calculation unit calculates the start address and the data block size of each scheduled slice data block according to the start address, the slice size, and the data block size of each input data block, and schedules input data from the outside. The output data address calculation unit calculates the start address and the data block size of each scheduled slice data block according to the start address, the slice size and the data block size of each output data block, and schedules and outputs the generated output data to the outside.
And each target output data scheduling unit outputs data outwards after calculating data with the size of one slice according to the block size and the slice size in the configuration information. And when the calculation of the size of the whole task data block is finished, the data is output outwards and the state information is fed back to the task scheduling unit. And the task scheduling unit sends a task response after monitoring that the erasure correcting task is completed.
Please refer to fig. 4, fig. 4 is a schematic diagram illustrating two consecutive task scheduling provided in an embodiment of the present application, and fig. 4 illustrates a working process of the hardware architecture by taking 2 erasure correction tasks processed consecutively as an example in a working manner of the hardware structure and the working dataflow proposed in the present application. As can be seen from the figure, 2 tasks can be continuously sent to the hardware processing unit, the throughput rate of the input data reaches a full load state, the 6 PE units will output data according to the number of blocks of the encoding and decoding output data of the tasks, and the system can be seen to generate the erasure-corrected output data in a small data slice rolling scheduling manner. As shown in fig. 4, the vertical axis represents hardware resources, and the horizontal axis represents time. The task request input task request 1 is a 7+3 code, the input data scheduling unit outputs data blocks 1-6, the matrix scheduling unit outputs 7 slices (each slice is 4Kbyte) of the coding matrix 1 every time, the configuration scheduling unit outputs task 1 configuration, the output scheduling unit 1-6 outputs 3 slices of the check blocks 1-3 every time, and the task responds to the output task 1 and outputs a response 7+3 code. The task request input task request 1 is decoded by 7+3, the input data scheduling unit outputs check blocks 1-6, the matrix scheduling unit outputs 7 slices (4 KByte per slice) of the decoding matrix 1 each time, the configuration scheduling unit outputs task 2 configuration, the output scheduling units 1-6 output 4 slices of the data blocks 1-4 each time, and the task responds to the response 7+3 code of the output task 2.
In the embodiment, for the RS erasure code under distributed storage, data slicing processing is performed on a large block size task through hardware, and erasure of a large data block is realized by dynamically scheduling small blocks of data in a rolling manner. The process has no perception to upper application software, and can be suitable for hardware systems cached on different chips in a configurable slicing size and hardware slicing mode. If the scheme of the invention is not adopted, when an erasure correcting task with a larger data block size (for example, the size of the erasure correcting data block is 2Mbyte), one way is to split the erasure correcting task with the larger data block into small erasure correcting tasks in a software layer, which increases the scheduling cost, because a large number of task requests and tasks are correspondingly transmitted in a hardware system, and the software overhead is also increased. Alternatively, a very large buffer size is added to each processing unit in hardware, for example, if the task is 2Mbyte in size, the buffer size of each processing unit is required to be 2Mbyte, which significantly increases on-chip cache resources and also puts a high demand on the buffer size of the system input unit.
An erasure task processing method provided in an embodiment of the present application may include the following steps:
determining a slice parameter corresponding to the erasure correcting task; wherein the slice parameters include: presetting a slice size, a data block size of the erasure correcting task, the number of input data blocks and the number of output data blocks;
calculating first address information of each input slice data block according to the slice parameters; wherein the first address information comprises a start address and a data size of the input slice data block;
calculating second address information of each output slice data block according to the slice parameters; wherein the second address information comprises a start address and a data size of the output slice data block;
and controlling the input and the output of an erasure processing module according to the first address information and the second address information so as to complete the erasure task.
In this embodiment, the slice parameter of the erasure correction task is determined, and the start address and the data size of each input slice data block can be determined according to the slice parameter, so that the DMA module controls the input of the erasure correction processing module according to the start address and the data size of each input slice data block. The present embodiment may determine the start address and the data size of each output slice data block according to the slice parameters, so that the DMA module controls the output of the erasure processing module according to the start address and the data size of each output slice data block. According to the scheme, data slicing processing is carried out on the data blocks of the erasure correcting task, and erasure correction of large data blocks is achieved by dynamically rolling and scheduling small data blocks. The process has no perception to upper application software, and can be suitable for hardware systems cached on different chips in a configurable slicing size and hardware slicing mode.
Further, determining the slice parameter corresponding to the erasure task includes:
setting the size of the preset slice according to the scheduling mode of the DMA unit and the caching capacity output by hardware;
or adaptively adjusting the size of the preset slice according to the current buffer vacancy condition.
Furthermore, the erasure correcting processing module comprises a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, an output data scheduling unit, a processing unit and an output data scheduling unit;
the task scheduling unit is used for receiving and analyzing the erasure correction task to obtain analysis information, and sending the analysis information to the data scheduling unit, the matrix scheduling unit, the output data scheduling unit and the data slice scheduling unit.
Further, the data scheduling unit is configured to input each input slice data block into the processing unit according to the number of input data blocks, the number of output data blocks, and the size of each input slice data block.
Further, the matrix scheduling unit is configured to determine an encoding matrix and/or a decoding matrix according to the number of the input data blocks and the number of the output data blocks, and input the determined encoding matrix and/or decoding matrix to the processing unit.
Further, the configuration scheduling unit is configured to generate configuration information according to the number of input data blocks, the number of output data blocks, and the size of each input slice data block, and input the configuration information into the processing unit and the output data scheduling unit.
Further, the output data scheduling unit is configured to determine whether the erasure correcting task is completed according to the configuration information; and the task scheduling unit is also used for returning response information to the task scheduling unit after the erasure task is completed.
Since the embodiment of the method portion corresponds to the embodiment of the system portion, please refer to the description of the embodiment of the system portion for the embodiment of the method portion, which is not repeated here.
The present application also provides a storage medium having a computer program stored thereon, which when executed, may implement the steps provided by the above-described embodiments. The storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The application further provides an electronic device, which may include a memory and a processor, where the memory stores a computer program, and the processor may implement the steps provided by the foregoing embodiments when calling the computer program in the memory. Of course, the electronic device may also include various network interfaces, power supplies, and the like.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

Claims (10)

1. An erasure task processing system, comprising:
the data slice scheduling unit is used for determining slice parameters corresponding to the erasure correcting tasks and sending the slice parameters to the input data address calculation unit and the output data address calculation unit; wherein the slice parameters include: presetting a slice size, a data block size of the erasure correcting task, the number of input data blocks and the number of output data blocks;
the input data address calculation unit is used for calculating first address information of each input slice data block according to the slice parameters and sending the first address information to the DMA unit; wherein the first address information comprises a start address and a data size of the input slice data block;
the output data address calculation unit is used for calculating second address information of each output slice data block according to the slice parameters and sending the second address information to the DMA unit; wherein the second address information comprises a start address and a data size of the output slice data block;
and the DMA unit is used for controlling the input and the output of the erasure correcting processing module according to the first address information and the second address information so as to complete the erasure correcting task.
2. An erasure task processing system according to claim 1, wherein the data slice scheduling unit includes:
the slice parameter determining subunit is used for setting the size of the preset slice according to the scheduling mode of the DMA unit and the caching capacity output by the hardware;
or, the parameter adaptive adjustment subunit is configured to adaptively adjust the size of the preset slice according to the current buffer idle condition.
3. The erasure correcting task processing system according to claim 1, wherein the erasure correcting processing module includes a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, an output data scheduling unit, a processing unit, and an output data scheduling unit;
the task scheduling unit is used for receiving and analyzing the erasure correction task to obtain analysis information, and sending the analysis information to the data scheduling unit, the matrix scheduling unit, the output data scheduling unit and the data slice scheduling unit.
4. An erasure task processing system according to claim 3, wherein the data scheduling unit is configured to input each of the input sliced data blocks into the processing unit according to the number of the input data blocks, the number of the output data blocks, and the size of each of the input sliced data blocks.
5. An erasure task processing system according to claim 3, wherein the matrix scheduling unit is configured to determine an encoding matrix and/or a decoding matrix according to the number of input data blocks and the number of output data blocks, and input the determined encoding matrix and/or decoding matrix into the processing unit.
6. A system according to claim 3, wherein the configuration scheduling unit is configured to generate configuration information according to the number of input data blocks, the number of output data blocks, and the size of each input slice data block, and input the configuration information into the processing unit and the output data scheduling unit.
7. The erasure task processing system of claim 6, wherein the output data scheduling unit is configured to determine whether the erasure task is completed according to the configuration information; and the task scheduling unit is also used for returning response information to the task scheduling unit after the erasure task is completed.
8. An erasure task processing method, comprising:
determining a slice parameter corresponding to the erasure correcting task; wherein the slice parameters include: presetting a slice size, a data block size of the erasure correcting task, the number of input data blocks and the number of output data blocks;
calculating first address information of each input slice data block according to the slice parameters; wherein the first address information comprises a start address and a data size of the input slice data block;
calculating second address information of each output slice data block according to the slice parameters; wherein the second address information comprises a start address and a data size of the output slice data block;
and controlling the input and the output of an erasure processing module according to the first address information and the second address information so as to complete the erasure task.
9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the erasure task processing method according to claim 8 when calling the computer program in the memory.
10. A storage medium having stored thereon computer-executable instructions which, when loaded and executed by a processor, perform the steps of the erasure task processing method of claim 8.
CN202111444997.6A 2021-11-30 2021-11-30 Erasure task processing system, method, electronic device and storage medium Pending CN114253684A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111444997.6A CN114253684A (en) 2021-11-30 2021-11-30 Erasure task processing system, method, electronic device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111444997.6A CN114253684A (en) 2021-11-30 2021-11-30 Erasure task processing system, method, electronic device and storage medium

Publications (1)

Publication Number Publication Date
CN114253684A true CN114253684A (en) 2022-03-29

Family

ID=80793642

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111444997.6A Pending CN114253684A (en) 2021-11-30 2021-11-30 Erasure task processing system, method, electronic device and storage medium

Country Status (1)

Country Link
CN (1) CN114253684A (en)

Similar Documents

Publication Publication Date Title
US10014881B2 (en) Multiple erasure codes for distributed storage
US10003357B2 (en) Systems and methods for verification of code resiliency for data storage
KR101862341B1 (en) Data storage device with data compression function
US11582299B2 (en) Allocating cache memory in a dispersed storage network
US9465692B2 (en) High reliability erasure code distribution
US10740198B2 (en) Parallel partial repair of storage
CN113687975B (en) Data processing method, device, equipment and storage medium
US10949129B2 (en) Adjusting data ingest based on compaction rate in a dispersed storage network
CN103729151A (en) Failure data recovery method based on improved erasure codes
CN113297000A (en) RAID (redundant array of independent disks) coding circuit and coding method
CN113505019A (en) Erasure code data and check recovery method, device, equipment and readable medium
CN104035732A (en) Data placing method aiming at erasure codes
US10528282B2 (en) Modifying and utilizing a file structure in a dispersed storage network
US20190238481A1 (en) Partial task execution in a dispersed storage network
CN116954523B (en) Storage system, data storage method, data reading method and storage medium
CN114253684A (en) Erasure task processing system, method, electronic device and storage medium
CN103942116A (en) Method for conducting fault-tolerance processing on data and data processing system
CN112286449B (en) RS erasure processing equipment and distributed storage system
CN115113816A (en) Erasure code data processing system, method, computer device and medium
CN113504874B (en) Load perception-based self-adaptive granularity erasure coding and decoding acceleration method and system
CN107463462B (en) Data restoration method and data restoration device
CN106302573B (en) Method, system and device for processing data by adopting erasure code
US20160043738A1 (en) Distribution And Replication of Erasure Codes
CN111858129A (en) Erasure code reading request processing method, system, equipment and computer medium
CN114879904B (en) Data storage erasure method, device, equipment and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination