WO2018018827A1 - Procédé de commande de stockage de données et serveur de données - Google Patents

Procédé de commande de stockage de données et serveur de données Download PDF

Info

Publication number
WO2018018827A1
WO2018018827A1 PCT/CN2016/108679 CN2016108679W WO2018018827A1 WO 2018018827 A1 WO2018018827 A1 WO 2018018827A1 CN 2016108679 W CN2016108679 W CN 2016108679W WO 2018018827 A1 WO2018018827 A1 WO 2018018827A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
storage
read
stored
fpga chip
Prior art date
Application number
PCT/CN2016/108679
Other languages
English (en)
Chinese (zh)
Inventor
张建
李发明
Original Assignee
深圳市中博科创信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市中博科创信息技术有限公司 filed Critical 深圳市中博科创信息技术有限公司
Publication of WO2018018827A1 publication Critical patent/WO2018018827A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
    • G06F11/1044—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices with specific ECC/EDC distribution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data

Definitions

  • the present invention relates to the field of data storage technologies, and in particular, to a data storage control method and a data server.
  • the erasure code is widely stored in the storage system, used for backup and storage management of data written and read in the storage system to ensure normal reading and writing and data security.
  • the erasure code has more and more computational data, so occupying more storage space of the storage system not only reduces the data storage capacity of the storage system, but also reduces the erasure code. The efficiency of the calculation.
  • the main object of the present invention is to provide a data storage control method and a data server, which aim to solve the problem of low data processing capability of the storage system.
  • the present invention provides a data storage control method, and the data storage control method includes:
  • the data server When the data server receives the first data, storing the first data in an FPGA chip;
  • the step of storing the second data into each storage unit of the memory comprises:
  • Each of the second data is sequentially called and stored in a corresponding storage unit
  • the second data that is called is stored in the corresponding storage unit.
  • the step of splitting the first data stored in the FPGA chip according to the storage ratio and the verification data to obtain the second data comprises:
  • the first data and the split check data are used as the second data.
  • the data storage control method further includes:
  • the erasure code in the FPGA chip is invoked according to the data read instruction
  • the step of recombining the data to be read stored in each of the storage units corresponding to the data read instruction according to the erasure code comprises:
  • the data to be read is reorganized according to the erasure code.
  • the present invention further provides a data storage control method, the data storage control method comprising the following steps:
  • the data server When the data server receives the first data, storing the first data in a high speed memory of the FPGA chip;
  • the erasure code stored in the FPGA chip is called to calculate the verification data of the first data stored in the FPGA chip, and the first data stored in the FPGA chip and the first data are calculated. Verify the storage ratio of the data;
  • the step of storing the second data in each storage unit of the memory comprises:
  • Each of the second data is sequentially called, and each of the second data is allocated a corresponding storage address according to a preset rule
  • the second data that is called is stored in the corresponding storage unit.
  • the data storage control method further includes:
  • the step of splitting the first data stored in the FPGA chip according to the storage ratio and the verification data to obtain the second data comprises:
  • the first data and the split check data are used as the second data.
  • the data storage control method further includes:
  • the erasure code in the FPGA chip is invoked according to the data read instruction
  • the step of recombining the data to be read stored in each of the storage units corresponding to the data read instruction according to the preset rule and the erasure code comprises:
  • the data to be read is reorganized according to the erasure code and the preset rule.
  • the data storage control method further includes:
  • the prompt information is output.
  • the present invention further provides a data server, where the data server includes:
  • a first storage module configured to store the first data in an FPGA chip when the data server receives the first data
  • a splitting module configured to calculate, after the first data receiving is completed, an erasure code stored in the FPGA chip, to calculate parity data of the first data stored in the FPGA chip, and calculate the stored in the FPGA chip First data and a storage ratio of the verification data;
  • a second storage module configured to split the first data stored in the FPGA chip and the verification data according to the storage ratio to obtain a plurality of second data, and store the second data into the memory In each storage unit.
  • the second storage module comprises:
  • a calling unit configured to sequentially call each of the second data, and store the data in a corresponding storage unit
  • a first determining unit configured to determine, when each of the second data is invoked, whether a storage unit corresponding to the second data to be stored has a storage failure
  • a sending unit configured to send an interrupt message to the storage unit corresponding to the second data, if the storage unit corresponding to the second data to be stored has a storage failure, so that the storage unit corresponding to the second data is The interrupt message stops storing the second data;
  • a storage unit configured to store the called second data into a corresponding storage unit if a storage failure corresponding to the storage unit corresponding to the second data to be stored does not occur.
  • the second storage module further includes:
  • a determining unit configured to determine, according to the storage ratio, a first split component of the first data stored in the FPGA chip and a second split component of the check data
  • a splitting unit configured to split the first data stored in the FPGA chip according to the first split component, and split the verification data according to the second split component, and split the split data
  • the first data stored in the FPGA chip and the split verification data are used as the second data.
  • the data server further includes:
  • Calling module when the data server receives the data read instruction, calling the erasure code in the FPGA chip according to the data read instruction;
  • a reassembly module configured to recombine data to be read stored in each of the storage units corresponding to the data read instruction according to the erasure code.
  • the reorganization module comprises:
  • a reading unit configured to sequentially read data to be read stored in each of the storage units corresponding to the data read instruction
  • a second determining unit configured to determine, after the reading of the data to be read is completed, whether there is invalid data in the read data to be read;
  • a first reorganization unit configured to recover the failure data by using the non-failure data in the data to be read when the failure data exists in the data to be read, and to the non-failure data according to the erasure code And recovering the failed data after the recovery;
  • a second recombining unit configured to recombine the data to be read according to the erasure code when there is no invalid data in the data to be read.
  • the first data is stored in the FPGA chip, and after the first data is received, the erasure code stored in the FPGA chip is called to calculate the first data stored in the FPGA chip. Verifying the data, calculating the storage ratio of the first data and the verification data stored in the FPGA chip, splitting the first data stored in the FPGA chip according to the storage ratio, and verifying the data to obtain a plurality of second data, and the second The data is stored separately in individual memory locations of the memory.
  • the erasure code is stored in the FPGA chip.
  • Data and a storage ratio of the first data and the verification data after the first data and the verification data are split according to the storage ratio to obtain a plurality of second data, the second data is separately stored in the memory
  • the resolution of the data by the erasure code, the determination of the check data, and the calculation of the storage ratio of the data and the check data are all performed in the FPGA chip, and the memory is only responsible for allocating the storage unit to store the split.
  • the divided data and the verification data can be used, thereby reducing the occupation of the memory running space of the erasure code, improving the data storage capability of the memory, and further improving the calculation efficiency of the erasure code.
  • FIG. 1 is a schematic flow chart of a first embodiment of a data storage control method according to the present invention
  • FIG. 2 is a schematic flowchart of a refinement step of storing the second data in each storage unit of the memory in the data storage control method of the present invention
  • FIG. 3 is a schematic flowchart of a refinement step of splitting the first data stored in the FPGA chip and the verification data to obtain a plurality of second data according to the storage ratio in the data storage control method of the present invention
  • FIG. 4 is a schematic flow chart of a second embodiment of a data storage control method according to the present invention.
  • FIG. 5 is a schematic flowchart of a refinement step of step S220 in the second embodiment shown in FIG. 4 according to the present invention
  • FIG. 6 is a schematic diagram of functional modules of a first embodiment of a data server according to the present invention.
  • FIG. 7 is a schematic diagram of a refinement function module of a second storage module in a second embodiment of a data server according to the present invention.
  • FIG. 8 is a schematic diagram of another refinement function module of the second storage module in the third embodiment of the data server of the present invention.
  • FIG. 9 is a schematic diagram of functional modules of a fourth embodiment of a data server according to the present invention.
  • FIG. 10 is a schematic diagram of a refinement function module of a reassembly module in a fifth embodiment of a data server according to the present invention.
  • the present invention provides a data storage control method.
  • FIG. 1 is a schematic flowchart diagram of a first embodiment of a data storage control method according to the present invention.
  • the data storage control method includes:
  • Step S110 when the data server receives the first data, storing the first data in the FPGA chip;
  • the erasure code is stored in the FPGA chip, and the FPGA chip is applied to the data storage server, and is independent of the CPU and the memory in the data server.
  • the data server receives the first data written externally, the data server temporarily stores the received first data in the high speed memory of the FPGA chip.
  • the data server includes a PC end, a smart phone, a smart TV, and a tablet computer; the high speed memory includes a solid state hard disk (Solid) State Disk, SSD).
  • Step S120 after the first data receiving is completed, calling the erasure code stored in the FPGA chip to calculate the verification data of the first data stored in the FPGA chip, and calculating the first data stored in the FPGA chip and the storage of the verification data.
  • the data server After determining that the first data has been received and completely stored in the high speed memory of the FPGA chip, the data server calls the erasure code stored in the FPGA chip, and the temporary storage is performed at the high speed according to the algorithm of the erasure code.
  • the verification data of the first data in the memory and calculating the storage ratio of the first data and the verification data in the high speed memory. For example, after the first data of a total of 3 bytes is completely stored in the high speed memory of the FPGA chip, the PC end calls the erasure code stored in the FPGA chip to calculate the check data of the first data as 1 byte, and then the calculation is performed.
  • the storage ratio of the first data and the check data in the high speed memory is 3:1.
  • the check data is obtained by performing check coding on the first data to obtain a character string. When part of the data in the first data is lost, the erasure code passes the check data and is not lost in the first data. The data recovers the lost data in the first data.
  • Step S130 splitting the first data and the verification data stored in the FPGA chip according to the storage ratio to obtain a plurality of second data, and storing the second data in each storage unit of the memory.
  • the data server is allocated from the local memory and the second data A unit to be stored with the same number of data, and the storage space size of each unit to be stored matches the byte size of the second data, and the second data is separately stored in each unit to be stored in the memory.
  • Each of the to-be-stored units can store only one second data. For example, after the storage ratio of the first data and the check data is 3:1, the first data is split into three data segments, each data segment is 1 byte, combined with one byte check. The data is obtained by 4 bytes of second data, and the PC side allocates 4 storage units from the memory of the local end, and the storage space of the 4 storage units is one byte, and the 4 bytes are the second. The data is stored in 4 storage units.
  • the check data is split into M segments
  • the first data is split into K segments
  • the total memory Y occupied by the M segments is equal to the total occupied by the K segments.
  • the first data when the first data is received by the data server, the first data is stored in the FPGA chip, and after the first data is received, the erasure code stored in the FPGA chip is called to calculate the first data stored in the FPGA chip.
  • the erasure code is stored in the FPGA chip.
  • the data server receives the first data
  • the first data is temporarily stored in the FPGA chip, and the erasure code stored in the FPGA chip is called to calculate the verification corresponding to the first data.
  • Data and a storage ratio of the first data and the verification data after the first data and the verification data are split according to the storage ratio to obtain a plurality of second data, the second data is separately stored in the memory
  • the resolution of the data by the erasure code, the determination of the check data, and the calculation of the storage ratio of the data and the check data are all performed in the FPGA chip, and the memory is only responsible for allocating the storage unit to store the split.
  • the divided data and the verification data can be used, thereby reducing the occupation of the memory running space of the erasure code, improving the data storage capability of the memory, and further improving the calculation efficiency of the erasure code.
  • FIG. 2 is a schematic flowchart of the refinement step of step S130 in the first embodiment of the present invention.
  • the storing the second data in each storage unit of the memory includes:
  • Step S131 sequentially calling each second data and storing it in a corresponding storage unit
  • the erasure code is given to the data server.
  • the to-be-storage unit sends a preparation message, where each preparation message includes a target second data that needs to be stored in each unit to be stored.
  • a preparation log is generated according to the preparation message.
  • the response message is sent to the FPGA chip.
  • the FPGA sends the storage error message to the storage unit to send a storage message, indicating that the storage unit is to be written into the preparation log.
  • the second data of a target is saved in the unit to be stored, and is used to temporarily save a target second data that needs to be stored by the to-be-stored unit.
  • each of the second data is allocated a storage address, and the storage address is allocated to each of the second data in a manner that the number of addresses is sequentially increased. .
  • the second data a, the second data b, the second data c, and the second data d are allocated storage addresses in such a manner that the number of addresses is sequentially increased, respectively, 0001, 0010, and 01001. 0100.
  • Step S132 determining, by each time the second data is called, determining whether a storage unit corresponding to the second data to be stored has a storage failure
  • Step S133 if yes, sending an interrupt message to the storage unit corresponding to the second data, so that the storage unit corresponding to the second data stops storing the second data according to the interrupt message;
  • the erasure code is invoked to send an interrupt message to the to-be-stored unit, and the to-be-stored unit deletes the preparation log according to the interruption message, and interrupts storage of the target second data, the target The second data is waiting for the storage, and if the storage failure recovery of the storage unit is detected within a preset time, the target second data is stored in the to-be-stored unit, and if the storage unit is not detected within a preset time The storage failure recovery, the data server re-allocates the to-be-stored unit that matches the target second data from the local storage, and stores the target second data into the to-be-stored unit.
  • Step S134 if no, storing the called second data into the corresponding storage unit.
  • the second data to be called is stored in the corresponding storage unit when it is determined that the storage unit does not have a storage failure.
  • each time the second data is called it is determined whether the storage unit corresponding to the second data to be stored has a storage failure, and if so, The storage unit corresponding to the second data sends an interrupt message, so that the storage unit corresponding to the second data stops storing the second data according to the interrupt message, and if not, stores the called second data in the corresponding storage unit. Since each storage unit of the memory may have a storage failure at any time during the process of storing the second data, detecting each storage unit of the memory by time, avoiding sending storage disorder during the process of storing the second data, resulting in data Lost to ensure data security.
  • FIG. 3 is a schematic flowchart of a refinement step of step S130 according to the first embodiment of the present invention, where the first data stored in the FPGA chip and the checksum are split according to the storage ratio.
  • the data gets multiple second data including:
  • Step S135 determining a first split component of the first data stored in the FPGA chip and a second split component of the check data according to the storage ratio;
  • Step S136 splitting the first data stored in the FPGA chip according to the first split component, and splitting the check data according to the second split component, and splitting the first data stored in the split FPGA chip and splitting The post-check data is used as the second data.
  • the first split component of the first data stored in the FPGA chip and the second split component of the check data are determined according to the storage ratio, and the first data stored in the FPGA chip is split according to the first split component. And splitting the verification data according to the second split component, and using the first data stored in the split FPGA chip and the split check data as the second data. Since the erasure code splits the entire segment of data into a small segment of data according to a certain storage ratio, it not only facilitates the storage of the split data, but also avoids the problem that the data in the memory cannot be recovered after the data is invalid.
  • the data storage control method includes:
  • Step S210 when the data server receives the data read instruction, invoking the erasure code in the FPGA chip according to the data read instruction;
  • Step S220 reorganizing the data to be read stored in each storage unit corresponding to the data read instruction according to the erasure code.
  • the erasure code stored in the FPGA chip is invoked, according to the erasure code.
  • the data redundancy function restores the data to be read into the entire data from the reorganization direction of the storage address from small to large.
  • the data redundancy function includes splitting the data into a plurality of segments according to a preset rule, and is capable of reconstructing the previous data according to the preset rule.
  • the erasure code in the FPGA chip is called according to the data read instruction, and the data to be read stored in each storage unit corresponding to the data read instruction is reorganized according to the erasure code.
  • the data to be read is reorganized by the erasure code, so that the data to be read can be read normally, which improves the fault tolerance of the memory.
  • FIG. 5 is a schematic flowchart of the refinement step of step S220 in the second embodiment of the present invention.
  • the refinement step of step S220 includes:
  • Step S221 sequentially reading data to be read stored in each storage unit corresponding to the data read instruction
  • Step S222 after the data to be read is read, determining whether there is invalid data in the read data to be read;
  • the erasure code after the erasure code in the FPGA chip is called according to the data read instruction, the erasure code sends a data call instruction to the data to be read stored in each storage unit corresponding to the data read instruction, Each storage unit corresponding to the data read instruction sends the locally stored data to be read to the FPGA chip according to the call instruction, and is temporarily stored in the high speed memory in the FPGA chip, and the data to be read is read. After completion, it is judged whether there is invalid data in the read data to be read.
  • Step S223 when there is invalid data in the data to be read, recovering the invalid data by using the non-failed data in the data to be read, and reconstructing the non-failed data and the restored invalid data according to the erasure code;
  • the erasure code When there is invalid data in the data to be read, and the number of the invalid data is less than a preset value, the erasure code recovers the invalid data through the non-failed data in the data to be read based on the data redundancy function, after restoring the invalid data, according to The data redundancy function in the erasure code restores the non-failed data and the recovered invalid data to the entire segment of data from the reorganization direction of the storage address from small to large.
  • the preset value is an integer smaller than the segment of the check data in the data to be read. If the number of the check data in the data to be read is K, the preset value is an integer smaller than K.
  • the erasure code cannot recover the invalid data through the non-failed data in the data to be read based on the data redundancy function, so the data to be read cannot be read normally. take.
  • Step S224 When there is no invalid data in the data to be read, the data to be read is reorganized according to the erasure code.
  • the data to be read is restored to the entire data according to the data redundancy function in the erasure code from the reorganization direction of the storage address from small to large.
  • the present embodiment sequentially reads the data to be read stored in each storage unit corresponding to the data read command, and after the data to be read is read, determines whether there is invalid data in the read data to be read, and is to be read.
  • the invalid data is recovered by using the unfailed data in the data to be read, and the non-failed data and the restored invalid data are reorganized according to the erasure code, and when there is no invalid data in the data to be read According to the erasure code, the data to be read is reorganized, so that the invalid data can still be read normally after recovery, thereby ensuring data security.
  • the invention further provides a data server.
  • FIG. 6 is a schematic diagram of functional modules of a first embodiment of a data server according to the present invention.
  • the data server includes: a first storage module 110, a split module 120, and a second storage module 130.
  • the first storage module 110 is configured to store the first data in the FPGA chip when the data server receives the first data
  • the erasure code is stored in the FPGA chip, and the FPGA chip is applied to the data storage server, and is independent of the CPU and the memory in the data server.
  • the data server temporarily stores the received first data in the high speed memory of the FPGA chip.
  • the data server includes a PC end, a smart phone, a smart TV, and a tablet computer; the high speed memory includes a solid state hard disk (Solid) State Disk, SSD).
  • the splitting module 120 is configured to, after the first data receiving is completed, invoke the erasure code stored in the FPGA chip to calculate the verification data of the first data stored in the FPGA chip, and calculate the first data stored in the FPGA chip. And the storage ratio of the verification data;
  • the splitting module 120 in the data server calls the erasure code stored in the FPGA chip, according to the The erasure code algorithm calculates the parity data of the first data temporarily stored in the high speed memory, and calculates the storage ratio of the first data and the parity data in the high speed memory. For example, after the first data of a total of 3 bytes is completely stored in the high speed memory of the FPGA chip, the PC end calls the erasure code stored in the FPGA chip to calculate the check data of the first data as 1 byte, and then the calculation is performed.
  • the storage ratio of the first data and the check data in the high speed memory is 3:1.
  • the check data is obtained by performing check coding on the first data to obtain a character string. When part of the data in the first data is lost, the erasure code passes the check data and is not lost in the first data. The data recovers the lost data in the first data.
  • the second storage module 130 is configured to split the first data and the verification data stored in the FPGA chip according to the storage ratio to obtain a plurality of second data, and store the second data in each storage unit of the memory.
  • the second storage module 130 splits the first data and the check data according to the storage ratio to obtain a plurality of second data, and the data server A unit to be stored is allocated in the memory of the end, and the size of the storage space of each unit to be stored matches the byte size of the second data, and the second data is separately stored in the memory.
  • Each of the to-be-stored units can store only one second data. For example, after the storage ratio of the first data and the check data is 3:1, the first data is split into three data segments, each data segment is 1 byte, combined with one byte check. The data is obtained by 4 bytes of second data, and the PC side allocates 4 storage units from the memory of the local end, and the storage space of the 4 storage units is one byte, and the 4 bytes are the second. The data is stored in 4 storage units.
  • the check data is split into M segments
  • the first data is split into K segments
  • the total memory Y occupied by the M segments is equal to the total occupied by the K segments.
  • the first data when the first data is received by the first storage module 110 in the data server, the first data is stored in the FPGA chip.
  • the split module 120 calls the correction stored in the FPGA chip. Deleting the code to calculate the verification data of the first data stored in the FPGA chip, and calculating the storage ratio of the first data and the verification data stored in the FPGA chip, and the second storage module 130 splits the stored in the FPGA chip according to the storage ratio.
  • a data and verification data obtain a plurality of second data, and the second data is separately stored in each storage unit of the memory.
  • the erasure code is stored in the FPGA chip.
  • the data server When the data server receives the first data, the first data is temporarily stored in the FPGA chip, and the erasure code stored in the FPGA chip is called to calculate the verification corresponding to the first data.
  • Data and a storage ratio of the first data and the verification data after the first data and the verification data are split according to the storage ratio to obtain a plurality of second data, the second data is separately stored in the memory
  • the resolution of the data by the erasure code, the determination of the check data, and the calculation of the storage ratio of the data and the check data are all performed in the FPGA chip, and the memory is only responsible for allocating the storage unit to store the split.
  • the divided data and the verification data can be used, thereby reducing the occupation of the memory running space of the erasure code, improving the data storage capability of the memory, and further improving the calculation efficiency of the erasure code.
  • the second storage module 130 includes a calling unit 131, a first determining unit 132, and a sending unit. 133 and storage unit 134.
  • the calling unit 131 is configured to sequentially call each second data and store the data in a corresponding storage unit;
  • the calling unit 131 invokes the The erasure code sends a preparation message to the to-be-stored unit, and each preparation message includes a target second data that needs to be stored in each to-be-stored unit.
  • the to-be-stored unit receives the preparation message, it prepares according to the preparation message.
  • Logging after the preparation log of the to-be-stored unit, sending a response message to the FPGA chip, when the FPGA receives the response message, invoking the erasure code to send a storage message to the to-be-stored unit, indicating that the to-be-stored unit writes Enter a target second data in the preparation log.
  • the preparation log is saved in the unit to be stored, and is used to temporarily save a target second data that needs to be stored by the to-be-stored unit.
  • each of the second data is allocated a storage address, and the storage address is allocated to each of the second data in a manner that the number of addresses is sequentially increased. .
  • the second data a, the second data b, the second data c, and the second data d are allocated storage addresses in such a manner that the number of addresses is sequentially increased, respectively, 0001, 0010, and 01001. 0100.
  • the first determining unit 132 is configured to determine, when each call to the second data, whether a storage unit corresponding to the second data to be stored has a storage failure;
  • the first determining unit 132 determines whether the unit to be stored corresponding to the second data to be stored feeds back a response message, and if receiving the response information fed back by the unit to be stored, determining The storage unit does not have a storage failure. If the response information with the storage unit feedback is not received, it is determined that the storage unit has a storage failure.
  • the sending unit 133 is configured to send an interrupt message to the storage unit corresponding to the second data if the storage unit corresponding to the second data to be stored has a storage failure, so that the storage unit corresponding to the second data stops storing according to the interrupt message.
  • Second data Second data
  • the sending unit 133 calls the erasure code to send an interrupt message to the to-be-stored unit, and the to-be-stored unit deletes the preparation log according to the interrupt message, and interrupts the target.
  • the second data is stored, and the target second data is waiting for storage. If the storage failure recovery of the storage unit is detected within a preset time, the target second data is stored in the to-be-stored unit if the preset time is If the storage failure recovery of the storage unit is not detected, the data server re-allocates the to-be-stored unit that matches the target second data from the local storage, and stores the target second data into the to-be-stored unit.
  • the storage unit 134 is configured to store the called second data into the corresponding storage unit if the storage unit corresponding to the second data to be stored does not have a storage failure.
  • the storage unit 134 stores the called second data into the corresponding storage unit.
  • each time the second data is called it is determined whether the storage unit corresponding to the second data to be stored has a storage failure, and if so, The storage unit corresponding to the second data sends an interrupt message, so that the storage unit corresponding to the second data stops storing the second data according to the interrupt message, and if not, stores the called second data in the corresponding storage unit. Since each storage unit of the memory may have a storage failure at any time during the process of storing the second data, detecting each storage unit of the memory by time, avoiding sending storage disorder during the process of storing the second data, resulting in data Lost to ensure data security.
  • the second storage module 130 includes a determining unit 135 and a splitting unit 136.
  • the determining unit 135 is configured to determine, according to the storage ratio, a first split component of the first data stored in the FPGA chip and a second split component of the check data;
  • the determining unit 135 determines that the storage ratio of the first data and the check data is M:K, determining that the first split component of the first data is M, and the second of the check data Split the component to K.
  • the splitting unit 136 is configured to split the first data stored in the FPGA chip according to the first split component, and split the check data according to the second split component, and store the split FPGA chip The first data and the split check data are used as the second data.
  • the splitting unit 136 splits the first data into M segments according to the first split component M, and splits the check data into K segments according to the second split component K, thereby forming M+K. Second data.
  • the first split component of the first data stored in the FPGA chip and the second split component of the check data are determined according to the storage ratio, and the first data stored in the FPGA chip is split according to the first split component. And splitting the verification data according to the second split component, and using the first data stored in the split FPGA chip and the split check data as the second data. Since the erasure code splits the entire segment of data into a small segment of data according to a certain storage ratio, it not only facilitates the storage of the split data, but also avoids the problem that the data in the memory cannot be recovered after the data is invalid.
  • the data server further includes: a calling module 210 and a recombining module 220.
  • the calling module 210 is configured to invoke an erasure code in the FPGA chip according to the data read instruction when the data server receives the data read instruction;
  • the recombination module 220 is configured to reassemble data to be read stored in each storage unit corresponding to the data read instruction according to the erasure code.
  • the calling module 210 calls the erasure code stored in the FPGA chip, and the recombination module 220:
  • the data redundancy function in the erasure code the data to be read is restored to the entire data segment from the reorganization direction of the storage address from small to large.
  • the data redundancy function includes splitting the data into a plurality of segments according to a preset rule, and is capable of reconstructing the previous data according to the preset rule.
  • the erasure code in the FPGA chip is called according to the data read instruction, and the data to be read stored in each storage unit corresponding to the data read instruction is reorganized according to the erasure code.
  • the data to be read is reorganized by the erasure code, so that the data to be read can be read normally, which improves the fault tolerance of the memory.
  • the reassembly module 220 includes a reading unit 221, a second judging unit 222, and a first reorganization. Unit 223 and second recombination unit 224.
  • the reading unit 221 is configured to sequentially read data to be read stored in each storage unit corresponding to the data read instruction;
  • the second determining unit 222 is configured to determine, after the data to be read is read, whether there is invalid data in the read data to be read;
  • the read unit 221 sends the erasure code to the data to be read stored in each storage unit corresponding to the data read command.
  • a data calling instruction and each storage unit corresponding to the data reading instruction sends the locally stored data to be read to the FPGA chip according to the calling instruction, and temporarily stores the data to be read in the high speed memory in the FPGA chip, to be read
  • the second determining unit 222 determines whether there is invalid data in the read data to be read.
  • the first recombining unit 223 is configured to recover invalid data in the data to be read when there is invalid data in the data to be read, and perform non-failure data and the restored invalid data according to the erasure code. Reorganization
  • the erasure code recovers the invalid data by using the non-failed data in the data to be read based on the data redundancy function. After restoring the invalid data, the un-failed data and the restored failed data are restored to the entire data according to the data redundancy function in the erasure code from the small to large recombination direction of the storage address.
  • the preset value is an integer smaller than the segment of the check data in the data to be read. If the number of the check data in the data to be read is K, the preset value is an integer smaller than K.
  • the erasure code cannot recover the invalid data through the non-failed data in the data to be read based on the data redundancy function, so the data to be read cannot be read normally. take.
  • the second recombining unit 224 is configured to reassemble the data to be read according to the erasure code when there is no invalid data in the data to be read.
  • the second recombining unit 224 restores the data to be read to the whole according to the data redundancy function in the erasure code from the reorganization direction of the storage address from small to large. Segment data.
  • the present embodiment sequentially reads the data to be read stored in each storage unit corresponding to the data read command, and after the data to be read is read, determines whether there is invalid data in the read data to be read, and is to be read.
  • the invalid data is recovered by using the unfailed data in the data to be read, and the non-failed data and the restored invalid data are reorganized according to the erasure code, and when there is no invalid data in the data to be read According to the erasure code, the data to be read is reorganized, so that the invalid data can still be read normally after recovery, thereby ensuring data security.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Human Computer Interaction (AREA)
  • Computer Security & Cryptography (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Techniques For Improving Reliability Of Storages (AREA)

Abstract

L'invention concerne un procédé de commande de stockage de données et un serveur de données. Le procédé comporte les étapes consistant à : quand un serveur de données reçoit des premières données, stocker les premières données dans une puce FPGA (S110) ; quand la réception des premières données est terminée, programmer un code de correction d'effacement stocké dans la puce FPGA pour calculer des données de vérification pour les premières données stockées dans la puce FPGA, et calculer un rapport de stockage des premières données stockées dans la puce FPGA par rapport aux données de vérification (S120) ; et diviser, selon le rapport de stockage, les premières données stockées dans la puce FPGA et les données de vérification pour obtenir une pluralité de deuxièmes données, et stocker respectivement les deuxièmes données dans diverses unités de stockage d'une mémoire (S130). Le procédé et le serveur de données réduisent l'occupation de l'espace de fonctionnement dans la mémoire au moyen du code correcteur d'effacement, améliore la capacité de stockage de données de la mémoire, et améliore ainsi l'efficacité de calcul du code correcteur d'effacement.
PCT/CN2016/108679 2016-07-25 2016-12-06 Procédé de commande de stockage de données et serveur de données WO2018018827A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610590235.XA CN106201766B (zh) 2016-07-25 2016-07-25 数据存储控制方法及数据服务器
CN201610590235.X 2016-07-25

Publications (1)

Publication Number Publication Date
WO2018018827A1 true WO2018018827A1 (fr) 2018-02-01

Family

ID=57494844

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/108679 WO2018018827A1 (fr) 2016-07-25 2016-12-06 Procédé de commande de stockage de données et serveur de données

Country Status (2)

Country Link
CN (1) CN106201766B (fr)
WO (1) WO2018018827A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111813549A (zh) * 2020-07-07 2020-10-23 河南城建学院 一种服务器资源配置方法
US11182249B1 (en) 2020-06-24 2021-11-23 International Business Machines Corporation Block ID encoding in an erasure coded storage system

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106598499A (zh) * 2016-12-14 2017-04-26 深圳市中博睿存科技有限公司 一种基于fpga的分布式文件系统架构
CN111386513B (zh) * 2018-05-03 2021-09-07 华为技术有限公司 数据处理的方法、装置和系统芯片
CN109491599A (zh) * 2018-10-24 2019-03-19 山东超越数控电子股份有限公司 一种分布式存储系统及其异构加速方法
CN113918378A (zh) * 2020-07-10 2022-01-11 华为技术有限公司 数据存储方法、存储系统、存储设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100293439A1 (en) * 2009-05-18 2010-11-18 David Flynn Apparatus, system, and method for reconfiguring an array to operate with less storage elements
CN102609378A (zh) * 2012-01-18 2012-07-25 中国科学院计算技术研究所 一种消息式内存访问装置及其访问方法
CN103631976A (zh) * 2013-05-30 2014-03-12 中国科学院电子学研究所 一种小型多通道串行数据记录器
CN105095013A (zh) * 2015-06-04 2015-11-25 华为技术有限公司 数据存储方法、恢复方法、相关装置以及系统
CN105359108A (zh) * 2013-08-05 2016-02-24 英特尔公司 带有自适应纠删码生成的存储系统
CN105791353A (zh) * 2014-12-23 2016-07-20 深圳市腾讯计算机系统有限公司 基于纠删码的分布式数据存储方法及系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100293439A1 (en) * 2009-05-18 2010-11-18 David Flynn Apparatus, system, and method for reconfiguring an array to operate with less storage elements
CN102609378A (zh) * 2012-01-18 2012-07-25 中国科学院计算技术研究所 一种消息式内存访问装置及其访问方法
CN103631976A (zh) * 2013-05-30 2014-03-12 中国科学院电子学研究所 一种小型多通道串行数据记录器
CN105359108A (zh) * 2013-08-05 2016-02-24 英特尔公司 带有自适应纠删码生成的存储系统
CN105791353A (zh) * 2014-12-23 2016-07-20 深圳市腾讯计算机系统有限公司 基于纠删码的分布式数据存储方法及系统
CN105095013A (zh) * 2015-06-04 2015-11-25 华为技术有限公司 数据存储方法、恢复方法、相关装置以及系统

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11182249B1 (en) 2020-06-24 2021-11-23 International Business Machines Corporation Block ID encoding in an erasure coded storage system
CN111813549A (zh) * 2020-07-07 2020-10-23 河南城建学院 一种服务器资源配置方法
CN111813549B (zh) * 2020-07-07 2023-07-25 河南城建学院 一种服务器资源配置方法

Also Published As

Publication number Publication date
CN106201766A (zh) 2016-12-07
CN106201766B (zh) 2018-03-20

Similar Documents

Publication Publication Date Title
WO2018018827A1 (fr) Procédé de commande de stockage de données et serveur de données
WO2019019378A1 (fr) Procédé et appareil de traitement de service, adaptateur et support de stockage lisible par ordinateur
WO2019019376A1 (fr) Procédé, appareil et dispositif de gestion de processus de service, et support d'informations lisible par ordinateur
WO2019041406A1 (fr) Dispositif, terminal et procédé de reconnaissance d'image indécente et support de stockage lisible par ordinateur
WO2015127859A1 (fr) Procédé et appareil de détection de texte sensible
WO2015035777A1 (fr) Procédé et système de mise à niveau de logiciel pour un terminal mobile
WO2018058919A1 (fr) Procédé, appareil, dispositif de génération d'informations d'identification, et support de stockage lisible par ordinateur
WO2015108283A1 (fr) Procédé de détection d'erreurs d'applications pour service de diffusion continue en nuage, et appareil et système associés
WO2019051890A1 (fr) Procédé et dispositif de commande de terminal et support de stockage lisible par ordinateur
WO2019019374A1 (fr) Procédé, appareil et système permettant de commander un appareil électroménager à l'aide d'un dispositif vocal intelligent
WO2017069508A1 (fr) Appareil de réception et procédé de décodage correspondant
WO2016082267A1 (fr) Procédé et système de reconnaissance vocale
WO2018120429A1 (fr) Procédé de mise à jour de ressources, terminal, support de stockage lisible par ordinateur et dispositif de mise à jour de ressources
WO2015007007A1 (fr) Procédé et dispositif permettant une correction automatique d'un convertisseur analogique-numérique
WO2018120457A1 (fr) Procédé de traitement de données, appareil, dispositif et support de stockage lisible par ordinateur
WO2019000801A1 (fr) Procédé, appareil et dispositif de synchronisation de données, et support d'informations lisible par ordinateur
WO2017107388A1 (fr) Procédé de commutation de version hdmi et dispositif d'affichage
WO2018233367A1 (fr) Procédé et appareil d'enregistrement de cas, terminal et support de stockage lisible par ordinateur
WO2015041475A1 (fr) Appareil de transmission et son procédé de perforation
WO2018166107A1 (fr) Procédé de compatibilité à base hybride, adaptateur, appareil et système d'exploitation, et support de stockage lisible par ordinateur
WO2017084302A1 (fr) Procédé destiné à la lecture de vidéo de démarrage d'un terminal d'affichage et terminal d'affichage
WO2018000856A1 (fr) Procédé de mise en œuvre de transfert de message de réseau de recouvrement sdn, terminal, appareil et support d'informations lisible par ordinateur
WO2018201697A1 (fr) Procédé, dispositif et équipement permettant de produire un fichier et support d'informations lisible par ordinateur
WO2015093870A1 (fr) Procédé et dispositif de gestion de données
WO2017032122A1 (fr) Procédé et appareil de détection d'un téléviseur numérique

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16910405

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 24.06.2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16910405

Country of ref document: EP

Kind code of ref document: A1