CN110990189A - Data storage method and device, electronic equipment and computer readable storage medium - Google Patents
Data storage method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN110990189A CN110990189A CN201911216677.8A CN201911216677A CN110990189A CN 110990189 A CN110990189 A CN 110990189A CN 201911216677 A CN201911216677 A CN 201911216677A CN 110990189 A CN110990189 A CN 110990189A
- Authority
- CN
- China
- Prior art keywords
- blocks
- data
- code information
- erasure code
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Security & Cryptography (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data storage method, a data storage device, an electronic device and a computer readable storage medium, relates to the technical field of data processing, and solves the technical problem that a distributed storage system is poor in flexibility. The method comprises the following steps: acquiring original data to be stored, and generating M data blocks and N check blocks based on the original data; respectively storing the M data blocks and the N check blocks into M + N storage blocks; persisting erasure code information into each of the memory blocks; wherein, the erasure code information includes the values of M and N.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data storage method and apparatus, an electronic device, and a computer-readable storage medium.
Background
In a distributed storage system, two of the most common data redundancy techniques are multi-copy policy and Erasure Code (EC). Compared with a multi-copy strategy, the erasure code has the advantages of lower redundancy, higher disk utilization rate and the like. For erasure codes, it should be noted that a plurality of original data blocks are calculated to obtain a plurality of check blocks, and for all the data blocks and check blocks, when any check block element is lost, the original data can be recovered through a corresponding algorithm.
In the existing distributed storage system, the configuration of the whole cluster is fixed, that is, the number of blocks of the data block and the number of blocks of the check block are both fixed, and once the cluster is built, the configuration cannot be changed, otherwise, data can be damaged, and reading cannot be performed.
For different copy requirements of users, the existing distributed storage system can only be realized by independently building a new cluster, and if the new cluster is not built independently, the existing distributed storage system cannot flexibly adapt to different copy requirements, so that the flexibility of the existing distributed storage system is poor.
Disclosure of Invention
The invention aims to provide a data storage method, a data storage device, an electronic device and a computer readable storage medium, so as to solve the technical problem that the existing distributed storage system is poor in flexibility.
In a first aspect, an embodiment of the present application provides a data storage method, which is applied to a distributed storage system, and the method includes:
acquiring original data to be stored, and generating M data blocks and N check blocks based on the original data;
respectively storing the M data blocks and the N check blocks into M + N storage blocks;
persisting erasure code information into each of the memory blocks;
wherein, the erasure code information includes the values of M and N.
In one possible implementation, the step of persisting erasure code information to each of the memory blocks includes:
and expanding the block ID of each storage block, and writing the erasure code information into the expanded block ID of each storage block.
In one possible implementation, the step of persisting erasure code information to each of the memory blocks includes:
generating a description file containing erasure code information;
and saving the description file to each storage block.
In one possible implementation, the step of persisting erasure code information to each of the memory blocks includes:
and writing the erasure code information into each data block and each check block.
In a second aspect, a data storage method is provided, which is applied to a distributed storage client, and the method includes:
acquiring erasure code information of data to be read;
searching M + N storage blocks matched with the erasure code information in a distributed storage system;
respectively reading M data blocks and N check blocks from M + N storage blocks;
wherein, the erasure code information includes the values of M and N.
In one possible implementation, the step of searching M + N memory blocks matching the erasure code information in the distributed storage system includes:
reading a block ID of each storage block in the distributed storage system, wherein the block ID comprises erasure code information;
and searching M + N storage blocks with the block IDs matched with the erasure code information of the data to be read.
In one possible implementation, the step of searching M + N memory blocks matching the erasure code information in the distributed storage system includes:
reading description information of each storage block in the distributed storage system, wherein the description information comprises erasure code information;
and searching M + N storage blocks of which the description information is matched with the erasure code information of the data to be read.
In one possible implementation, the step of searching M + N memory blocks matching the erasure code information in the distributed storage system includes:
reading a data block or a check block stored in each storage block in the distributed storage system, wherein the data block or the check block comprises erasure code information;
and searching M + N storage blocks matched with the erasure code information of the data to be read.
In a third aspect, there is provided a data storage apparatus for use in a distributed storage system, the apparatus comprising:
the acquisition module is used for acquiring original data to be stored;
a generating module, configured to generate M data blocks and N check blocks based on the original data;
a storage module, configured to store the M data blocks and the N check blocks into M + N storage blocks, respectively;
the persistence module is used for persisting erasure code information to each storage block;
wherein, the erasure code information includes the values of M and N.
In a fourth aspect, a data storage apparatus is provided, which is applied to a distributed storage client, and includes:
the acquisition unit is used for acquiring erasure code information of data to be read;
the searching unit is used for searching M + N storage blocks matched with the erasure code information in the distributed storage system;
a reading unit, configured to read M data blocks and N check blocks from M + N storage blocks, respectively;
wherein, the erasure code information includes the values of M and N.
In a fifth aspect, this application provides an electronic device, which includes a memory and a processor, where the memory stores a computer program that is executable on the processor, and the processor executes the computer program to implement the method of the first aspect or the second aspect.
In a sixth aspect, this embodiment of the present application further provides a computer-readable storage medium storing machine executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method of the first or second aspect.
The embodiment of the application brings the following beneficial effects:
in the data storage method, the data storage device, the electronic device and the computer-readable storage medium provided by the embodiments of the present application, first, original data to be stored is obtained, and M data blocks and N check blocks are generated based on the original data, then, storing the M data blocks and the N check blocks into M + N storage blocks respectively, and then persisting erasure code information including the M-value and the N-value to each of the memory blocks, the number M of the data blocks and the number N of the check blocks are persisted into the information of each storage block, so that the user terminal can write in the corresponding storage blocks according to different requirements during selection, therefore, different flexible configurations of a plurality of different data blocks and check blocks can be supported in one cluster according to different copy requirements of users, so that the distributed object storage system has better expandability and flexibility.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the detailed description of the present application or the technical solutions in the prior art, the drawings needed to be used in the detailed description of the present application or the prior art description will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a data storage method according to an embodiment of the present application;
fig. 2 is a flowchart illustrating an example of a user side in a data storage method according to an embodiment of the present disclosure;
FIG. 3 is a schematic flow chart illustrating another data storage method according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data storage device according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of another data storage device according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "comprising" and "having," and any variations thereof, as referred to in the embodiments of the present application, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
At present, for a plurality of different copy requirements of a user, a distributed storage system can only be realized by adopting a mode of independently building a new cluster, so that the distributed storage system cannot flexibly adapt to the plurality of different copy requirements, and the flexibility of the existing distributed storage system is poor.
The embodiment of the application provides a data storage method and device, electronic equipment and a computer readable storage medium. The method can solve the technical problem of poor flexibility of the conventional distributed storage system.
Embodiments of the present invention are further described below with reference to the accompanying drawings.
Fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present application. Wherein the method is applied to a distributed storage system. As shown in fig. 1, the method includes:
s110, acquiring original data to be stored, and generating M data blocks and N check blocks based on the original data.
In this step, the distributed storage system may encode the original data block by using an erasure code technique to obtain a check block through an erasure code algorithm, and store the data block and the check block together to achieve the purpose of fault tolerance.
For example, the distributed storage system calculates M blocks of original data to be stored to obtain N check blocks, and for the M + N blocks, when any N block elements are lost, the original data can be restored through a corresponding algorithm. It should be noted that the process of generating the parity block is encoding (ENCODE), and the process of recovering the missing data block is decoding (DECODE).
And S120, respectively storing the M data blocks and the N check blocks into M + N storage blocks.
In this step, the distributed storage system stores the M data blocks in step S110 and the N check blocks in step S110 into M + N storage blocks, respectively.
S130, the erasure code information is persisted to each storage block.
Wherein, the erasure code information includes the values of M and N. It should be noted that erasure codes can create a mathematical function to describe a set of numbers so that their accuracy can be checked and, once one of the numbers is lost, recovered.
In this step, the erasure code information is information recording these erasure codes, which includes the value of the number M of data blocks in step S110 and the value of the number N of check blocks in step S110.
The data storage method provided by the embodiment of the application can be used as a method for utilizing the data erasure codes of the distributed object storage system, and the support for different data blocks and check blocks is added on the basis of the existing erasure codes. Specifically, for each written storage block, the number of data blocks and the number of check blocks are persisted into the information of each storage block, and the client can write into the corresponding storage block according to different requirements when selecting. Therefore, different flexible configurations of a plurality of different data blocks and different check blocks can be supported in one cluster for different copy requirements of users, so that the distributed object storage system has better expandability and flexibility. Moreover, the data storage method provided by the embodiment of the application has low cost for compatible change of historical data, so that seamless compatibility of the existing data is realized, and the flexibility of the distributed storage system is greatly improved.
The above steps are described in detail below.
In some embodiments, before the step S130, the following steps may be further included:
and generating erasure code information according to the numerical values of M and N.
For erasure coded data blocks and parity blocks, various coding schemes may be employed. Illustratively, binary numbers or decimal numbers with different formats, such as erasure code information generated by the configuration of M + N, can be formed by combining M plus N.
Therefore, the user can keep the ID of the original storage block unchanged, and only the different M + N configuration of each storage block is sensed on the user side level, so that the overall operation efficiency is improved.
In some embodiments, the step S130 may include various implementations.
As an example, the step S130 may include the steps of:
and expanding the block ID of each storage block, and writing erasure code information into the expanded block ID of each storage block.
For example, under the existing common architecture, for each written minimum storage unit Block, the ID of its Block is increased by two bytes, where one byte represents the value of a data Block and the other byte represents the value of a check Block, and this information is persisted to the storage blocks of the disk.
In the embodiment of the application, on the existing basis, the support for different data blocks and check blocks is added, namely for each written minimum storage unit Block, the number of the data blocks and the check blocks is persisted into the information of the Block, so that a user terminal (Client) can write into the corresponding Block according to different requirements during selection.
Therefore, the minimum granularity supported by different configurations is Block, and a plurality of different data blocks and check blocks can be supported in one cluster, so that the distributed storage system has better expandability and flexibility.
Moreover, for the data of the stock, the upgrading program is compatible, the data format is changed into the format after the expansion, and meanwhile, the user side upgrades the logic and is compatible with the ID of the Block after the expansion. Therefore, for the user, the ID of the original Block can be kept unchanged, and only the different M + N configuration of each Block is sensed on the user side layer, so as to facilitate the operation of the user side.
Based on this, the erasure code information can be 16-bit binary number.
For example, the ID of the previous Block is abcdefg, and as shown in fig. 2, the coding of the erasure code is 8 for the data Block and 4 for the parity Block. Then M is 00001000 (binary representation of 8), N is 00000100 (binary representation of 4), MN is 0000100000000100 (binary representation of 2052) in step S110, and thus the ID of the expanded Block is 2052# abcdefg.
The existing Block Map structure uniformly stores all Block information. In this embodiment of the application, the structure is split and expanded into different sub-structures, and the encoding information of the expanded erasure code is a key, for example, 2052 described above. All blocks encoded in this mode are placed under this key at 2052. As shown in fig. 2, if the user at the Client (Client) selects the erasure code scheme of 8+4, the blocks of B1, B2 and B3 can be selected from this point for the user to write; if the user selects the 8+3 erasure code approach, then blocks of B4, B5, and B6 may be selected from this for the user to write. Therefore, the operation process of the user terminal can be more convenient.
As another example, the erasure code information is not necessarily stored in the ID of Block, but may be stored in other related structures. For example, the step S130 may include the steps of:
generating a description file containing erasure code information;
and saving the description file to each storage block.
Therefore, the erasure code information can be stored in the description file of the erasure code information, and the description file storing the erasure code information is saved in each storage block, so as to ensure the operating efficiency of the persistence process in step S130.
As another example, there may be other implementations besides storing the information of M + N in the ID of Block. For example, the step S130 may further include the steps of:
erasure code information is written into each data block and each parity block.
In practical applications, the persistent process may also write the erasure code information into a header (header) of the data block, and may be used for performing double check, triple check, or more multiple checks to ensure logical consistency during encoding and decoding.
Fig. 3 is a schematic flowchart of another data storage method according to an embodiment of the present application. The method is applied to the distributed storage client. As shown in fig. 3, the method includes:
s310, acquiring erasure code information of data to be read.
In this step, the distributed storage client may obtain erasure code information of the data to be read from the storage block in which the erasure code information is stored.
S320, searching M + N storage blocks matched with the erasure code information in the distributed storage system.
The distributed storage system stores a plurality of storage blocks.
In this step, the distributed storage client finds M + N storage blocks that match the erasure code information in step S310 from the multiple storage blocks stored in the distributed storage system. Wherein, the erasure code information includes the values of M and N.
S330, respectively reading M data blocks and N check blocks from the M + N storage blocks.
It should be noted that erasure codes can create a mathematical function to describe a set of numbers so that their accuracy can be checked and, once one of the numbers is lost, recovered. In this step, the erasure code information is information recording these erasure codes, which includes the value of the number M of data blocks and the value of the number N of check blocks.
By the data storage method provided by the embodiment of the application, the distributed storage client can write in the corresponding storage block according to different requirements during selection. Therefore, different flexible configurations of a plurality of different data blocks and different check blocks can be supported in one cluster for different copy requirements of users, so that the distributed storage client has better expandability and flexibility. Moreover, the data storage method provided by the embodiment of the application has low cost for compatible change of the historical data, so that seamless compatibility of the existing data is realized, and the use flexibility of the distributed storage client is greatly improved.
The above steps are described in detail below.
In some embodiments, step S320 may include various implementations.
As an example, the step S320 may include the steps of:
reading a block ID of each storage block in the distributed storage system, wherein the block ID comprises erasure code information;
and searching M + N storage blocks with the block IDs matched with the erasure code information of the data to be read.
Through the method provided by the step, the user can keep the ID of the original storage block unchanged, and only the distributed storage client side senses different M + N configurations of each storage block, so that the overall operation efficiency is improved.
As another example, the step S320 may include the steps of:
reading description information of each storage block in the distributed storage system, wherein the description information comprises erasure code information;
and searching M + N storage blocks of which the description information is matched with the erasure code information of the data to be read.
Therefore, the erasure code information can be stored in the description file of the erasure code information, and the distributed storage client can search the erasure code information from the description file of the storage block. And the erasure code information is stored by using the description file, so that the operating efficiency of the persistence process can be ensured.
As another example, the step S320 may further include the steps of:
reading a data block or a check block stored in each storage block in the distributed storage system, wherein the data block or the check block comprises erasure code information;
and searching M + N storage blocks matched with erasure code information of the data to be read.
In practical applications, erasure code information may also be stored in each data block and each check block, for example, erasure code information is set in a header of a data block, and can be used for performing double check, triple check, or more multiple checks, so as to ensure logical consistency during encoding and decoding.
The data storage method applied to the distributed storage client provided by the embodiment of the application has the same technical characteristics as the data storage method applied to the distributed storage system provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
FIG. 4 provides a schematic diagram of a data storage device. The data storage device can be applied to a distributed storage system. As shown in fig. 4, the data storage device 400 includes:
an obtaining module 401, configured to obtain original data to be stored;
a first generating module 402, configured to generate M data blocks and N check blocks based on original data;
a saving module 403, configured to save the M data blocks and the N check blocks into M + N storage blocks respectively;
a persistence module 404 for persisting erasure code information into each memory block;
wherein, the erasure code information includes the values of M and N.
In some embodiments, the apparatus further comprises:
and the second generation module is used for generating erasure code information according to the numerical values of M and N.
In some embodiments, the persistence module comprises:
the expansion submodule is used for expanding the block ID of each storage block;
and the writing sub-module is used for writing the erasure code information into the expanded block ID of each storage block.
In some embodiments, the persistence module comprises:
the generating submodule is used for generating a description file containing erasure code information;
and the saving submodule is used for saving the description file to each storage block.
In some embodiments, the persistence module comprises:
and the writing submodule is used for writing the erasure code information into each data block and each check block.
In some embodiments, the erasure code information is a 16-bit binary number.
The data storage device provided by the embodiment of the application has the same technical characteristics as the data storage method provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
FIG. 5 provides a schematic diagram of another data storage device. The data storage device can be applied to distributed storage clients. As shown in fig. 5, the data storage device 500 includes:
an obtaining unit 501, configured to obtain erasure code information of data to be read;
a searching unit 502, configured to search M + N storage blocks matched with erasure code information in a distributed storage system;
a reading unit 503, configured to read M data blocks and N check blocks from M + N storage blocks, respectively;
wherein, the erasure code information includes the values of M and N.
In some embodiments, the lookup unit 502 is specifically configured to:
reading a block ID of each storage block in the distributed storage system, wherein the block ID comprises erasure code information;
and searching M + N storage blocks with the block IDs matched with the erasure code information of the data to be read.
In some embodiments, the lookup unit 502 is further configured to:
reading description information of each storage block in the distributed storage system, wherein the description information comprises erasure code information;
and searching M + N storage blocks of which the description information is matched with the erasure code information of the data to be read.
In some embodiments, the lookup unit 502 is further configured to:
reading a data block or a check block stored in each storage block in the distributed storage system, wherein the data block or the check block comprises erasure code information;
and searching M + N storage blocks matched with erasure code information of the data to be read.
The data storage device provided by the embodiment of the application has the same technical characteristics as the data storage method provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
As shown in fig. 6, an electronic device 600 includes a memory 601 and a processor 602, where the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the method provided in the foregoing embodiment.
Referring to fig. 6, the electronic device further includes: a bus 603 and a communication interface 604, the processor 602, the communication interface 604 and the memory 601 being connected by the bus 603; the processor 602 is used to execute executable modules, such as computer programs, stored in the memory 601.
The Memory 601 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is implemented through at least one communication interface 604 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like may be used.
The bus 603 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
The memory 601 is used for storing a program, and the processor 602 executes the program after receiving an execution instruction, and the method performed by the apparatus defined by the process disclosed in any of the foregoing embodiments of the present application may be applied to the processor 602, or implemented by the processor 602.
The processor 602 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 602. The Processor 602 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 601, and the processor 602 reads the information in the memory 601 and performs the steps of the above method in combination with the hardware thereof.
Corresponding to the data storage method, the embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores machine executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to execute the steps of the data storage method.
The data storage device provided by the embodiment of the present application may be specific hardware on the device, or software or firmware installed on the device, etc. The device provided by the embodiment of the present application has the same implementation principle and technical effect as the foregoing method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the foregoing method embodiments where no part of the device embodiments is mentioned. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the foregoing systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the mobile control method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus once an item is defined in one figure, it need not be further defined and explained in subsequent figures, and moreover, the terms "first", "second", "third", etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope disclosed in the present application; such modifications, changes or substitutions do not depart from the scope of the embodiments of the present application. Are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (12)
1. A data storage method is applied to a distributed storage system, and the method comprises the following steps:
acquiring original data to be stored, and generating M data blocks and N check blocks based on the original data;
respectively storing the M data blocks and the N check blocks into M + N storage blocks;
persisting erasure code information into each of the memory blocks;
wherein, the erasure code information includes the values of M and N.
2. The method of claim 1, wherein the step of persisting erasure code information into each of the memory blocks comprises:
writing the erasure code information into a block ID of each of the memory blocks.
3. The method of claim 1, wherein the step of persisting erasure code information into each of the memory blocks comprises:
generating a description file containing erasure code information;
and saving the description file to each storage block.
4. The method of claim 1, wherein the step of persisting erasure code information into each of the memory blocks comprises:
and writing the erasure code information into each data block and each check block.
5. A data storage method is applied to a distributed storage client, and comprises the following steps:
acquiring erasure code information of data to be read;
searching M + N storage blocks matched with the erasure code information in a distributed storage system;
respectively reading M data blocks and N check blocks from M + N storage blocks;
wherein, the erasure code information includes the values of M and N.
6. The method of claim 5, wherein the step of searching the distributed storage system for the M + N storage blocks matching the erasure code information comprises:
reading a block ID of each storage block in the distributed storage system, wherein the block ID comprises erasure code information;
and searching M + N storage blocks with the block IDs matched with the erasure code information of the data to be read.
7. The method of claim 5, wherein the step of searching the distributed storage system for the M + N storage blocks matching the erasure code information comprises:
reading description information of each storage block in the distributed storage system, wherein the description information comprises erasure code information;
and searching M + N storage blocks of which the description information is matched with the erasure code information of the data to be read.
8. The method of claim 5, wherein the step of searching the distributed storage system for the M + N storage blocks matching the erasure code information comprises:
reading a data block or a check block stored in each storage block in the distributed storage system, wherein the data block or the check block comprises erasure code information;
and searching M + N storage blocks matched with the erasure code information of the data to be read.
9. A data storage apparatus, applied to a distributed storage system, the apparatus comprising:
the acquisition module is used for acquiring original data to be stored;
a generating module, configured to generate M data blocks and N check blocks based on the original data;
a storage module, configured to store the M data blocks and the N check blocks into M + N storage blocks, respectively;
the persistence module is used for persisting erasure code information to each storage block;
wherein, the erasure code information includes the values of M and N.
10. A data storage device, for use with a distributed storage client, the device comprising:
the acquisition unit is used for acquiring erasure code information of data to be read;
the searching unit is used for searching M + N storage blocks matched with the erasure code information in the distributed storage system;
a reading unit, configured to read M data blocks and N check blocks from M + N storage blocks, respectively;
wherein, the erasure code information includes the values of M and N.
11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and wherein the processor implements the steps of the method of any of claims 1 to 8 when executing the computer program.
12. A computer readable storage medium having stored thereon machine executable instructions which, when invoked and executed by a processor, cause the processor to execute the method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911216677.8A CN110990189A (en) | 2019-11-29 | 2019-11-29 | Data storage method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911216677.8A CN110990189A (en) | 2019-11-29 | 2019-11-29 | Data storage method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110990189A true CN110990189A (en) | 2020-04-10 |
Family
ID=70089368
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911216677.8A Pending CN110990189A (en) | 2019-11-29 | 2019-11-29 | Data storage method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110990189A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256472A (en) * | 2020-10-20 | 2021-01-22 | 平安科技（深圳）有限公司 | Distributed data calling method and device, electronic equipment and storage medium |
| CN112558881A (en) * | 2020-12-18 | 2021-03-26 | 上海七牛信息技术有限公司 | Method and system for migrating storage system |
| CN112988683A (en) * | 2021-02-07 | 2021-06-18 | 北京金山云网络技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113626649A (en) * | 2021-08-02 | 2021-11-09 | Oppo广东移动通信有限公司 | Data storage method, data storage device, storage medium and electronic equipment |
| CN117240873A (en) * | 2023-11-08 | 2023-12-15 | 阿里云计算有限公司 | Cloud storage system, data reading and writing method, device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150236725A1 (en) * | 2014-02-20 | 2015-08-20 | Quantum Corporation | Dynamically Configuring Erasure Code Redundancy And Distribution |
| CN106227731A (en) * | 2016-07-06 | 2016-12-14 | 联想(北京)有限公司 | A kind of data processing method, device and distributed file system |
| CN106776146A (en) * | 2016-12-29 | 2017-05-31 | 华为技术有限公司 | A kind of data verification method, apparatus and system |
| CN107589916A (en) * | 2017-09-29 | 2018-01-16 | 郑州云海信息技术有限公司 | A kind of entangling based on correcting and eleting codes deletes the creation method and relevant apparatus in pond |
| CN109117292A (en) * | 2017-06-22 | 2019-01-01 | 中兴通讯股份有限公司 | A kind of method and apparatus and cluster storage system of cluster-based storage |
-
2019
- 2019-11-29 CN CN201911216677.8A patent/CN110990189A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150236725A1 (en) * | 2014-02-20 | 2015-08-20 | Quantum Corporation | Dynamically Configuring Erasure Code Redundancy And Distribution |
| CN106227731A (en) * | 2016-07-06 | 2016-12-14 | 联想(北京)有限公司 | A kind of data processing method, device and distributed file system |
| CN106776146A (en) * | 2016-12-29 | 2017-05-31 | 华为技术有限公司 | A kind of data verification method, apparatus and system |
| CN109117292A (en) * | 2017-06-22 | 2019-01-01 | 中兴通讯股份有限公司 | A kind of method and apparatus and cluster storage system of cluster-based storage |
| CN107589916A (en) * | 2017-09-29 | 2018-01-16 | 郑州云海信息技术有限公司 | A kind of entangling based on correcting and eleting codes deletes the creation method and relevant apparatus in pond |
Non-Patent Citations (1)
| Title |
|---|
| （美）格雷（GRAY，J）: "《事务处理：概念与技术》", 北京：机械工业出版社, pages: 674 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256472A (en) * | 2020-10-20 | 2021-01-22 | 平安科技（深圳）有限公司 | Distributed data calling method and device, electronic equipment and storage medium |
| WO2021189905A1 (en) * | 2020-10-20 | 2021-09-30 | 平安科技（深圳）有限公司 | Distributed data retrieval method and apparatus, and electronic device and storage medium |
| CN112558881A (en) * | 2020-12-18 | 2021-03-26 | 上海七牛信息技术有限公司 | Method and system for migrating storage system |
| CN112988683A (en) * | 2021-02-07 | 2021-06-18 | 北京金山云网络技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113626649A (en) * | 2021-08-02 | 2021-11-09 | Oppo广东移动通信有限公司 | Data storage method, data storage device, storage medium and electronic equipment |
| CN117240873A (en) * | 2023-11-08 | 2023-12-15 | 阿里云计算有限公司 | Cloud storage system, data reading and writing method, device and storage medium |
| CN117240873B (en) * | 2023-11-08 | 2024-03-29 | 阿里云计算有限公司 | Cloud storage system, data reading and writing method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110990189A (en) | Data storage method and device, electronic equipment and computer readable storage medium | |
| CN108154211B (en) | Two-dimensional code generation method, two-dimensional code generation device, two-dimensional code service processing equipment and two-dimensional code | |
| CN113297000B (en) | RAID (redundant array of independent disks) coding circuit and coding method | |
| CN110083312B (en) | Disk capacity expansion method and device and computer equipment | |
| CA3177662C (en) | Data-storage method and apparatus, and data-recovery method and apparatus | |
| CN105468302A (en) | Data processing method, apparatus and system | |
| CN110545106A (en) | Method and device for coding time series data | |
| CN111353002B (en) | Training method and device for network representation learning model, electronic equipment and medium | |
| CN104965835A (en) | Method and apparatus for reading and writing files of a distributed file system | |
| CN112702167A (en) | Cluster feature code obtaining method and device and electronic equipment | |
| CN113468118B (en) | File increment storage method, device and storage medium based on blockchain | |
| CN112988458A (en) | Data backup method and device, electronic equipment and storage medium | |
| CN111857603B (en) | Data processing method and related device | |
| CN108614749B (en) | Data processing method and device | |
| CN114048061A (en) | Check block generation method and device | |
| US10505672B2 (en) | FEC decoding apparatus and method | |
| CN113253934A (en) | Flash bad block processing method and device, computer equipment and readable storage medium | |
| CN110704232B (en) | Method, device and equipment for repairing failed node in distributed system | |
| CN110209347B (en) | Traceable data storage method | |
| CN112579141A (en) | Firmware updating method, device, equipment and storage medium of embedded system | |
| CN110968255A (en) | Data processing method, data processing device, storage medium and processor | |
| CN113704200A (en) | Data storage method, device, equipment and storage medium | |
| CN112657196A (en) | Resource updating method and device, computer equipment and readable storage medium | |
| CN109144767B (en) | Data storage system and method of operating the same | |
| CN112269665A (en) | Memory processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |