CN111026580A - Data reading method, device, equipment and storage medium - Google Patents
Data reading method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111026580A CN111026580A CN201911204918.7A CN201911204918A CN111026580A CN 111026580 A CN111026580 A CN 111026580A CN 201911204918 A CN201911204918 A CN 201911204918A CN 111026580 A CN111026580 A CN 111026580A
- Authority
- CN
- China
- Prior art keywords
- data
- data blocks
- block set
- blocks
- data block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012795 verification Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004883 computer application Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/91—Television signal processing therefor
- H04N5/93—Regeneration of the television signal or of selected parts thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The application discloses a data reading method, which comprises the following steps: when target data is to be read, K data blocks corresponding to the target data are selected to form a data block set; reading each data block in the data block set; if the K data blocks are read successfully, restoring the target data and returning the target data to the upper layer application so that the upper layer application outputs the target data; if the data blocks are not successfully read, eliminating the data blocks which are not successfully read from the data block set, selecting the data blocks from the data blocks which are not selected and adding the data blocks into the data block set, and repeatedly executing the step of reading each data block in the data block set until the K + M data blocks corresponding to the target data are all selected. By applying the technical scheme provided by the embodiment of the application, the probability of successful reading is increased, and the stability and the fluency of video playing can be ensured. The application also discloses a data reading device, equipment and a storage medium, which have corresponding technical effects.
Description
Technical Field
The present application relates to the field of computer application technologies, and in particular, to a data reading method, apparatus, device, and storage medium.
Background
With the rapid development of computer technology, the application of storage systems is becoming more and more extensive, especially in the video monitoring industry, the bottom layer data is usually written in the storage system, and video playing is performed by reading in the upper layer application.
In a storage system, when data at the bottom layer is damaged due to an exception, such as an object file on an xfs (open source log file system) is damaged, data is unreadable due to a disk damage, a disk or a host node fails and exceeds a fault domain, and the like, a storage process cannot read the data. It is common practice to return to the upper layer application-the EIO or the storage process exits directly. The upper layer application also interrupts the service in the face of this situation.
In the video monitoring industry, when a small amount of data cannot be read, the playing of a video picture is not influenced, and the video picture is only displayed by a small part of screen-lost or cannot be perceived. But if the storage process returns to-EIO or exits, the video playing will be blocked, thereby affecting the stability and fluency of the video playing which is more concerned by the user.
In summary, how to effectively read data and ensure the stability and smoothness of video playing is a technical problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The application aims to provide a data reading method, a data reading device, data reading equipment and a storage medium, so as to ensure the stability and the fluency of video playing.
In order to solve the technical problem, the application provides the following technical scheme:
a data reading method comprising:
when target data are to be read, selecting K data blocks from K + M data blocks corresponding to the target data in a storage system to form a data block set, wherein the data in the storage system are stored in an erasure code redundancy mode, each data is composed of K source data blocks and M verification data blocks, and K, M is a positive integer;
reading each data block in the set of data blocks;
if the K data blocks in the data block set are read successfully, restoring the target data based on the data block set, and returning the target data to an upper layer application so that the upper layer application outputs the target data;
if the data blocks in the K data blocks of the data block set are not successfully read, eliminating the data blocks which are not successfully read from the data block set, selecting a corresponding number of data blocks from the unselected data blocks, adding the data blocks into the data block set, and repeatedly executing the step of reading each data block in the data block set until the K + M data blocks corresponding to the target data are all selected.
In one embodiment of the present application, the method further includes:
if a corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and K data blocks in the data block set are read successfully, taking the K data blocks in the data block set as source data blocks, and calculating to obtain M check data blocks;
and writing the K data blocks and the M check data blocks back to cover the original K + M data blocks corresponding to the target data.
In one embodiment of the present application, the method further includes:
and if the data blocks in the K data blocks of the data block set are not successfully read and K + M data blocks corresponding to the target data are all selected, returning preset data to the upper-layer application.
In a specific embodiment of the present application, the preset data is all 0 data.
In a specific embodiment of the present application, in a case that a data block is not successfully read in K data blocks of the data block set, and K + M data blocks corresponding to the target data are all selected, the method further includes:
and recording the identification information of the target data.
In a specific embodiment of the present application, in a case that a data block is not successfully read in K data blocks of the data block set, and K + M data blocks corresponding to the target data are all selected, the method further includes:
and deleting the K + M data blocks corresponding to the target data in the storage system.
A data reading apparatus comprising:
the device comprises a set forming module, a data block set generating module and a data block reading module, wherein the set forming module is used for selecting K data blocks from K + M data blocks corresponding to target data in a storage system to form a data block set when the target data are to be read, the data in the storage system are stored in an erasure code redundancy mode, each data is formed by K source data blocks and M verification data blocks, and K, M is a positive integer;
a data block reading module, configured to read each data block in the data block set;
a data returning module, configured to restore the target data based on the data block set and return the target data to an upper application if all the K data blocks in the data block set are successfully read, so that the upper application outputs the target data;
and the data reselection module is used for eliminating the data blocks which are not successfully read from the data block set if the data blocks in the K data blocks of the data block set are not successfully read, selecting a corresponding number of data blocks from the data blocks which are not successfully read, adding the data blocks into the data block set, and triggering the data block reading module to repeatedly execute the step of reading each data block in the data block set until the K + M data blocks corresponding to the target data are all selected.
In a specific embodiment of the present application, the apparatus further includes a data write-back module, configured to:
if a corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and K data blocks in the data block set are read successfully, taking the K data blocks in the data block set as source data blocks, and calculating to obtain M check data blocks;
and writing the K data blocks and the M check data blocks back to cover the original K + M data blocks corresponding to the target data.
A data reading apparatus comprising:
a memory for storing a computer program;
a processor for implementing the steps of any of the above data reading methods when executing the computer program.
A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps of the data reading method of any of the above.
By applying the technical scheme provided by the embodiment of the application, when target data are required to be read, K data blocks are selected from K + M data blocks corresponding to target data of a storage system to form a data block set, each data block in the data block set is read, if the K data blocks are read successfully, the target data are restored based on the data block set and returned to an upper layer application, the upper layer application outputs the target data, if the data blocks are not read successfully, the data blocks which are not read successfully are removed from the data block set, corresponding data blocks are selected from the data blocks which are not selected and added into the data block set, and the step of reading each data block in the data block set is repeatedly executed until the K + M data blocks corresponding to the target data are selected. And carrying out erasure code redundant storage on the bottom data, and restoring the target data as long as a group of K data blocks are successfully read, so that the probability of successful reading is increased, and the stability and the fluency of video playing can be ensured.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flowchart illustrating an implementation of a data reading method according to an embodiment of the present disclosure;
FIG. 2 is a flowchart illustrating another embodiment of a data reading method according to the present application;
FIG. 3 is a schematic structural diagram of a data reading apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data reading apparatus in an embodiment of the present application.
Detailed Description
The core of the application is to provide a data reading method, which can be applied to a storage process, wherein the storage process reads data in a storage system and returns the data to an upper application so as to facilitate the upper application to output the data.
The embodiment of the application stores the bottom layer data in a storage system in an erasure code redundancy mode. Erasure codes are a data redundancy technique that generally uses a K + M erasure rule. When writing data, data is firstly divided into K data blocks, then M data blocks are calculated, and finally the K + M data blocks are stored in a disk of a storage system. When reading, the original data can be calculated and restored from any K of the K + M data blocks.
When target data reading is needed, K data blocks are selected from K + M data blocks corresponding to target data of a storage system to form a data block set, each data block in the data block set is read, if the K data blocks are read successfully, the target data are restored based on the data block set and returned to an upper layer application, the upper layer application outputs the target data, if one data block is not read successfully, the data block which is not read successfully is removed from the data block set, a corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and the step of reading each data block in the data block set is repeatedly executed until the K + M data blocks corresponding to the target data are selected. And carrying out erasure code redundant storage on the bottom data, and restoring the target data as long as a group of K data blocks are successfully read, so that the probability of successful reading is increased, and the stability and the fluency of video playing can be ensured.
The upper layer application may be video surveillance software.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, a flowchart of an implementation of a data reading method provided in an embodiment of the present application is shown, where the method may include the following steps:
s110: when target data is to be read, K data blocks are selected from K + M data blocks corresponding to the target data of the storage system to form a data block set.
Data in the storage system is stored in an erasure code redundancy mode, each data is composed of K source data blocks and M verification data blocks, and K, M is a positive integer.
In the embodiment of the application, data in the storage system is stored in an erasure code redundancy mode, each data in the storage system is composed of K source data blocks and M verification data blocks, and K and M are positive integers.
When the target data is to be read, K data blocks may be selected from K + M data blocks corresponding to the target data in the storage system, and the selected K data blocks may include the source data block and may also include the check data block, which form a data block set.
The target data is data that needs to be read currently, such as a video frame or a segment of a video frame.
S120: each data block in the set of data blocks is read.
After a data block set is formed by selecting K data blocks from K + M data blocks corresponding to target data of the storage system, each data block in the data block set can be read, and whether reading is successful or not is determined.
S130: and if the K data blocks in the data block set are read successfully, restoring the target data based on the data block set, and returning the target data to the upper-layer application so that the upper-layer application outputs the target data.
Upon reading each data block in the set of data blocks, it may be determined whether each data block was read successfully. If the K data blocks in the data block set are read successfully, the target data can be restored based on the K data blocks in the data block set, and the obtained target data is returned to the upper-layer application, so that the upper-layer application can output the target data.
S140: if the data blocks in the K data blocks of the data block set are not successfully read, eliminating the data blocks which are not successfully read from the data block set, selecting a corresponding number of data blocks from the unselected data blocks, adding the data blocks into the data block set, and repeatedly executing the step of reading each data block in the data block set until the K + M data blocks corresponding to the target data are all selected.
Upon reading each data block in the set of data blocks, it may be determined whether each data block was read successfully. If one of the K data blocks in the data block set is not successfully read, the target data cannot be restored by using the K data blocks in the current data block set. In this case, the data blocks that have not been successfully read may be removed from the data block set, and a corresponding number of data blocks may be selected from the data blocks that have not been selected and added to the data block set. The updated data block set still contains K data blocks.
After the data block set is updated, the operation of reading each data block in the data block set in step S120 is repeatedly performed. Similarly, if the K data blocks in the data block set are read successfully, target data are restored based on the data block set and returned to the upper-layer application, so that the upper-layer application outputs the target data; if the data blocks in the K data blocks of the data block set are not successfully read, eliminating the data blocks which are not successfully read from the data block set, selecting a corresponding number of data blocks from the unselected data blocks, adding the data blocks into the data block set, and repeating the step of reading each data block in the data block set until K + M data blocks corresponding to the target data are all selected, so as to determine that the target data cannot be read, as shown in fig. 1.
The following are given by way of example, where K is 5 and M is 3:
when target data is to be read, 5 data blocks corresponding to the target data of the storage system form a data block set. Each data block in the set of data blocks is read. If 2 data blocks in 5 data blocks of the data block set are not successfully read, the 2 data blocks which are not successfully read are removed from the data block set, and 2 data blocks are selected from 3 data blocks which are not selected and added into the data block set. Each data block in the set of data blocks is read. If 1 data block in 5 data blocks of the data block set is not successfully read at this time, the 1 data block which is not successfully read is removed from the data block set, and 1 data block which is not selected is left, and the data block is added into the data block set. Each data block in the set of data blocks is re-read. And if the 5 data blocks in the data block set are read successfully at the moment, restoring the target data based on the data block set, and returning the target data to the upper-layer application so that the upper-layer application outputs the target data. If the 5 data blocks in the data block set are not successfully read, but the 5+3 data blocks corresponding to the target data are all selected, it may be determined that the target data cannot be read.
The above operation is performed for each data to be read, so that the consistency of the finally output data can be ensured, and the stability and the fluency of video playing can be ensured.
By applying the method provided by the embodiment of the application, when target data reading is needed, K data blocks are selected from K + M data blocks corresponding to target data of a storage system to form a data block set, each data block in the data block set is read, if the K data blocks are read successfully, the target data is restored based on the data block set and returned to an upper layer application, the upper layer application outputs the target data, if the data blocks are not read successfully, the data blocks which are not read successfully are removed from the data block set, corresponding data blocks are selected from the data blocks which are not selected and added into the data block set, and the step of reading each data block in the data block set is repeatedly executed until the K + M data blocks corresponding to the target data are selected. And carrying out erasure code redundant storage on the bottom data, and restoring the target data as long as a group of K data blocks are successfully read, so that the probability of successful reading is increased, and the stability and the fluency of video playing can be ensured.
In one embodiment of the present application, the method may further comprise the steps of:
if the corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and K data blocks in the data block set are read successfully, taking the K data blocks in the data block set as source data blocks, and calculating to obtain M check data blocks;
and writing the K data blocks and the M check data blocks back to original K + M data blocks corresponding to the coverage target data.
In the embodiment of the application, when target data is to be read, K data blocks are selected from K + M data blocks corresponding to target data of a storage system to form a data block set, each data block in the data block set is read, if the K data blocks are all read successfully, the target data is restored based on the data block set and returned to an upper layer application, the upper layer application outputs the target data, if data blocks are not read successfully, the data blocks which are not read successfully are removed from the data block set, corresponding data blocks are selected from the data blocks which are not selected and added into the data block set, and the step of reading each data block in the data block set is repeatedly executed until the K + M data blocks corresponding to the target data are all selected.
If the data blocks with the corresponding number are selected from the data blocks which are not selected, and after the data blocks are added into the data block set, the K data blocks in the data block set are read successfully, it is indicated that the damaged data blocks exist in the K + M data blocks corresponding to the target data, and the target data can be restored based on the K data blocks in the current data block set. In this case, K data blocks in the current data block set are used as source data blocks, M check data blocks are obtained by calculation, and the K data blocks and the M check data blocks are written back to cover the original K + M data blocks corresponding to the target data. Thus, when the target data needs to be read again, the probability that the selected K data blocks are successfully read is increased.
In one embodiment of the present application, the method may further comprise the steps of:
and if the data blocks in the K data blocks of the data block set are not successfully read and the K + M data blocks corresponding to the target data are all selected, returning preset data to the upper-layer application.
In the embodiment of the application, when target data is to be read, K data blocks are selected from K + M data blocks corresponding to target data of a storage system to form a data block set, each data block in the data block set is read, if the K data blocks are all read successfully, the target data is restored based on the data block set and returned to an upper layer application, the upper layer application outputs the target data, if data blocks are not read successfully, the data blocks which are not read successfully are removed from the data block set, corresponding data blocks are selected from the data blocks which are not selected and added into the data block set, and the step of reading each data block in the data block set is repeatedly executed until the K + M data blocks corresponding to the target data are all selected.
If data blocks in the K data blocks of the data block set are not successfully read and K + M data blocks corresponding to the target data are all selected, it is indicated that a group of K data blocks cannot be selected from the K + M data blocks corresponding to the target data to restore the target data. In this case, preset data may be returned to the upper layer application. And the upper layer application outputs the received preset data.
The preset data may be all-0 data, but may be other data. The upper layer application receives the preset data and then outputs the preset data, so that the interruption of service caused by the return-EIO or direct exit of the storage process is avoided, and the storage process can continue to read the next data, so that the upper layer application continuously receives the data and continuously outputs the data, and the video playing has stability and smoothness.
In an embodiment of the present application, in a case that a data block of K data blocks of a data block set is not successfully read, and K + M data blocks corresponding to target data are all selected, identification information of the target data, such as information of a start position, an end position, and the like of each data block corresponding to the target data, may be recorded. Thus, the technician can repair the data according to the importance degree of the target data through the recorded information.
And under the condition that data blocks in the K data blocks of the data block set are not successfully read and K + M data blocks corresponding to the target data are all selected, the K + M data blocks corresponding to the target data can be deleted in the storage system. In this case, it may be determined that the target data cannot be restored based on the K + M data blocks corresponding to the target data, and the K + M data blocks corresponding to the target data may be deleted in the storage system without leaving these data blocks. Therefore, the storage space can be saved, and when the target data needs to be read again, the preset data can be directly returned to the upper layer application, so that the data reading time is saved, and the data reading efficiency is improved.
For ease of understanding, the embodiments of the present application will be described by taking a specific example as shown in fig. 2 as an example.
The upper layer application is video monitoring software, the video monitoring software initiates a data reading request to the storage system, and a storage process in the storage system selects and reads K data blocks from K + M data blocks corresponding to target data. And determining whether each data block is successfully read, if so, restoring target data, returning the target data to the video monitoring software, and if not, removing the data blocks which are failed to be read, and supplementing and reading the data blocks which are not selected yet. And determining whether K data blocks which can be read successfully exist, if so, performing erasure correction calculation again, calculating M verification data blocks according to the K data blocks, then writing back the new K + M data blocks to cover the original K + M data blocks, and if not, deleting the K + M data blocks corresponding to the target data and returning all 0 data to the upper-layer application. The stability and the fluency of video playing performed by the video monitoring software can be ensured.
According to the embodiment of the application, the abnormal data are read, backfilled and returned to the full-0 data for the storage system can not report errors and quit when the bottom data is abnormal, so that the normal operation of the upper layer video monitoring service is ensured, the situation of pause or interruption of playing can not occur, and the applicability of the storage system in the video monitoring industry is enhanced.
Corresponding to the above method embodiments, the present application further provides a data reading apparatus, and the data reading apparatus described below and the data reading method described above may be referred to correspondingly.
Referring to fig. 3, the apparatus comprises the steps of:
a set forming module 310, configured to select K data blocks from K + M data blocks corresponding to target data of a storage system to form a data block set when the target data is to be read, where data in the storage system is stored in an erasure code redundancy manner, each data is formed by K source data blocks and M verification data blocks, and K, M is a positive integer;
a data block reading module 320, configured to read each data block in the data block set;
the data returning module 330 is configured to, if all the K data blocks in the data block set are successfully read, restore the target data based on the data block set, and return the target data to the upper-layer application, so that the upper-layer application outputs the target data;
the data reselection module 340 is configured to, if a data block in the K data blocks of the data block set is not successfully read, remove the unsuccessfully read data block from the data block set, select a corresponding number of data blocks from the unselected data blocks, add the data blocks into the data block set, and trigger the data block reading module to repeatedly execute the step of reading each data block in the data block set until K + M data blocks corresponding to the target data are all selected.
By applying the device provided by the embodiment of the application, when target data reading is needed, K data blocks are selected from K + M data blocks corresponding to target data of a storage system to form a data block set, each data block in the data block set is read, if the K data blocks are read successfully, the target data is restored based on the data block set and returned to an upper layer application, the upper layer application outputs the target data, if any data block is not read successfully, the data block which is not read successfully is removed from the data block set, a corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and the step of reading each data block in the data block set is repeatedly executed until the K + M data blocks corresponding to the target data are all selected. And carrying out erasure code redundant storage on the bottom data, and restoring the target data as long as a group of K data blocks are successfully read, so that the probability of successful reading is increased, and the stability and the fluency of video playing can be ensured.
In a specific embodiment of the present application, the apparatus further includes a data write-back module, configured to:
if the corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and K data blocks in the data block set are read successfully, taking the K data blocks in the data block set as source data blocks, and calculating to obtain M check data blocks;
and writing the K data blocks and the M check data blocks back to original K + M data blocks corresponding to the coverage target data.
In one embodiment of the present application, the data returning module 320 is further configured to:
and if the data blocks in the K data blocks of the data block set are not successfully read and the K + M data blocks corresponding to the target data are all selected, returning preset data to the upper-layer application.
In one embodiment of the present application, the preset data is all 0 data.
In a specific embodiment of the present application, the system further includes an information recording module, configured to:
and recording the identification information of the target data under the condition that the data blocks in the K data blocks of the data block set are not successfully read and the K + M data blocks corresponding to the target data are all selected.
In a specific embodiment of the present application, the system further includes a data deleting module, configured to:
and deleting the K + M data blocks corresponding to the target data in the storage system under the condition that the data blocks in the K data blocks of the data block set are not successfully read and the K + M data blocks corresponding to the target data are all selected.
Corresponding to the above method embodiment, an embodiment of the present application further provides a data reading apparatus, including:
a memory for storing a computer program;
and the processor is used for realizing the steps of the data reading method when executing the computer program.
As shown in fig. 4, which is a schematic diagram of a structure of a data reading apparatus, the data reading apparatus may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through a communication bus 13.
In the embodiment of the present application, the processor 10 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, etc.
The processor 10 may call a program stored in the memory 11, and in particular, the processor 10 may perform operations in an embodiment of the data reading method.
The memory 11 is used for storing one or more programs, the program may include program codes, the program codes include computer operation instructions, in this embodiment, the memory 11 stores at least the program for implementing the following functions:
when target data are to be read, selecting K data blocks from K + M data blocks corresponding to the target data of the storage system to form a data block set, storing the data in the storage system in an erasure code redundancy mode, wherein each data is composed of K source data blocks and M verification data blocks, and K, M is a positive integer;
reading each data block in the data block set;
if the K data blocks in the data block set are read successfully, restoring target data based on the data block set, and returning the target data to the upper-layer application so that the upper-layer application outputs the target data;
if the data blocks in the K data blocks of the data block set are not successfully read, eliminating the data blocks which are not successfully read from the data block set, selecting a corresponding number of data blocks from the unselected data blocks, adding the data blocks into the data block set, and repeatedly executing the step of reading each data block in the data block set until the K + M data blocks corresponding to the target data are all selected.
In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function (such as a sound playing function and an image playing function), and the like; the storage data area may store data created during use, such as read result data.
Further, the memory 11 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.
The communication interface 13 may be an interface of a communication module for connecting with other devices or systems.
Of course, it should be noted that the structure shown in fig. 4 does not constitute a limitation to the data reading device in the embodiment of the present application, and in practical applications, the data reading device may include more or less components than those shown in fig. 4, or some components may be combined.
Corresponding to the above method embodiments, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the data reading method.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The principle and the implementation of the present application are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A data reading method, comprising:
when target data are to be read, selecting K data blocks from K + M data blocks corresponding to the target data in a storage system to form a data block set, wherein the data in the storage system are stored in an erasure code redundancy mode, each data is composed of K source data blocks and M verification data blocks, and K, M is a positive integer;
reading each data block in the set of data blocks;
if the K data blocks in the data block set are read successfully, restoring the target data based on the data block set, and returning the target data to an upper layer application so that the upper layer application outputs the target data;
if the data blocks in the K data blocks of the data block set are not successfully read, eliminating the data blocks which are not successfully read from the data block set, selecting a corresponding number of data blocks from the unselected data blocks, adding the data blocks into the data block set, and repeatedly executing the step of reading each data block in the data block set until the K + M data blocks corresponding to the target data are all selected.
2. The method of claim 1, further comprising:
if a corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and K data blocks in the data block set are read successfully, taking the K data blocks in the data block set as source data blocks, and calculating to obtain M check data blocks;
and writing the K data blocks and the M check data blocks back to cover the original K + M data blocks corresponding to the target data.
3. The method of claim 1 or 2, further comprising:
and if the data blocks in the K data blocks of the data block set are not successfully read and K + M data blocks corresponding to the target data are all selected, returning preset data to the upper-layer application.
4. The method of claim 3, wherein the preset data is all 0 data.
5. The method of claim 3, wherein in a case that a data block of the K data blocks of the data block set is not successfully read and K + M data blocks corresponding to the target data are all selected, the method further comprises:
and recording the identification information of the target data.
6. The method of claim 3, wherein in a case that a data block of the K data blocks of the data block set is not successfully read and K + M data blocks corresponding to the target data are all selected, the method further comprises:
and deleting the K + M data blocks corresponding to the target data in the storage system.
7. A data reading apparatus, comprising:
the device comprises a set forming module, a data block set generating module and a data block reading module, wherein the set forming module is used for selecting K data blocks from K + M data blocks corresponding to target data in a storage system to form a data block set when the target data are to be read, the data in the storage system are stored in an erasure code redundancy mode, each data is formed by K source data blocks and M verification data blocks, and K, M is a positive integer;
a data block reading module, configured to read each data block in the data block set;
a data returning module, configured to restore the target data based on the data block set and return the target data to an upper application if all the K data blocks in the data block set are successfully read, so that the upper application outputs the target data;
and the data reselection module is used for eliminating the data blocks which are not successfully read from the data block set if the data blocks in the K data blocks of the data block set are not successfully read, selecting a corresponding number of data blocks from the data blocks which are not successfully read, adding the data blocks into the data block set, and triggering the data block reading module to repeatedly execute the step of reading each data block in the data block set until the K + M data blocks corresponding to the target data are all selected.
8. The apparatus of claim 7, further comprising a data write back module to:
if a corresponding number of data blocks are selected from the data blocks which are not selected and added into the data block set, and K data blocks in the data block set are read successfully, taking the K data blocks in the data block set as source data blocks, and calculating to obtain M check data blocks;
and writing the K data blocks and the M check data blocks back to cover the original K + M data blocks corresponding to the target data.
9. A data reading apparatus, characterized by comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data reading method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data reading method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911204918.7A CN111026580A (en) | 2019-11-29 | 2019-11-29 | Data reading method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911204918.7A CN111026580A (en) | 2019-11-29 | 2019-11-29 | Data reading method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111026580A true CN111026580A (en) | 2020-04-17 |
Family
ID=70207291
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911204918.7A Withdrawn CN111026580A (en) | 2019-11-29 | 2019-11-29 | Data reading method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026580A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108762976A (en) * | 2018-05-30 | 2018-11-06 | 郑州云海信息技术有限公司 | A kind of method, apparatus and storage medium reading correcting and eleting codes data |
| CN109271360A (en) * | 2018-08-03 | 2019-01-25 | 北京城市网邻信息技术有限公司 | Distributed objects storing data redundancy approach, device, equipment and storage medium |
| CN109412755A (en) * | 2018-11-05 | 2019-03-01 | 东方网力科技股份有限公司 | A kind of multimedia data processing method, device and storage medium |
| CN109491968A (en) * | 2018-11-13 | 2019-03-19 | 浙江鲸腾网络科技有限公司 | A kind of document handling method, device, equipment and computer readable storage medium |
| CN109558086A (en) * | 2018-12-03 | 2019-04-02 | 浪潮电子信息产业股份有限公司 | Data reading method, system and related components |
| CN109597566A (en) * | 2017-09-30 | 2019-04-09 | 杭州海康威视系统技术有限公司 | A kind of reading data, storage method and device |
-
2019
- 2019-11-29 CN CN201911204918.7A patent/CN111026580A/en not_active Withdrawn
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109597566A (en) * | 2017-09-30 | 2019-04-09 | 杭州海康威视系统技术有限公司 | A kind of reading data, storage method and device |
| CN108762976A (en) * | 2018-05-30 | 2018-11-06 | 郑州云海信息技术有限公司 | A kind of method, apparatus and storage medium reading correcting and eleting codes data |
| CN109271360A (en) * | 2018-08-03 | 2019-01-25 | 北京城市网邻信息技术有限公司 | Distributed objects storing data redundancy approach, device, equipment and storage medium |
| CN109412755A (en) * | 2018-11-05 | 2019-03-01 | 东方网力科技股份有限公司 | A kind of multimedia data processing method, device and storage medium |
| CN109491968A (en) * | 2018-11-13 | 2019-03-19 | 浙江鲸腾网络科技有限公司 | A kind of document handling method, device, equipment and computer readable storage medium |
| CN109558086A (en) * | 2018-12-03 | 2019-04-02 | 浪潮电子信息产业股份有限公司 | Data reading method, system and related components |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442473B (en) | Nonvolatile data storage method and device, electronic equipment and medium | |
| CN110597779A (en) | Data reading and writing method in distributed file system and related device | |
| CN109582488B (en) | Error prevention method and related device for solid state disk | |
| CN102135925A (en) | Method and device for detecting error check and correcting memory | |
| CN115620799A (en) | Firmware simulation-based fault site backtracking method and equipment and readable storage medium | |
| WO2021027271A1 (en) | Bad block information protection method and apparatus, computer device and storage medium | |
| CN115543693B (en) | Data recovery method and related equipment | |
| CN115599589B (en) | Data recovery method and related device | |
| CN111104256A (en) | Data reading method, device, equipment and storage medium | |
| CN111026580A (en) | Data reading method, device, equipment and storage medium | |
| CN109032536B (en) | Data storage method, device, system and equipment based on distributed cluster system | |
| CN111273862A (en) | Data storage method and device, readable medium and system thereof | |
| CN114138566B (en) | Data storage method and device of virtual machine, virtual machine and storage medium | |
| CN115878381A (en) | Data recovery method and device based on SRM disc, storage medium and electronic device | |
| US20100169572A1 (en) | Data storage method, apparatus and system for interrupted write recovery | |
| CN110658993A (en) | Snapshot rollback method, device, equipment and storage medium | |
| CN115599607A (en) | Data recovery method of RAID array and related device | |
| CN111026332B (en) | SSD bad block information protection method, SSD bad block information protection device, computer equipment and storage medium | |
| CN112307022A (en) | Metadata repairing method and related device | |
| CN112463444A (en) | Data inconsistency repairing method and related device | |
| CN111292796B (en) | RAID damage detailed information acquisition method, system, terminal and storage medium | |
| CN111475109B (en) | Data processing method, device, equipment and medium | |
| CN114879916B (en) | Method and device for managing storage unit | |
| CN113126927B (en) | Data deleting method and related device | |
| CN112463460B (en) | Erasure code-based small block data recovery method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200417 |
|
| WW01 | Invention patent application withdrawn after publication |