CN110347640B - File storage method and device, storage medium and electronic equipment - Google Patents
File storage method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN110347640B CN110347640B CN201910563487.7A CN201910563487A CN110347640B CN 110347640 B CN110347640 B CN 110347640B CN 201910563487 A CN201910563487 A CN 201910563487A CN 110347640 B CN110347640 B CN 110347640B
- Authority
- CN
- China
- Prior art keywords
- file
- storage medium
- target
- files
- erasure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/122—File system administration, e.g. details of archiving or snapshots using management policies
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure relates to a file storage method, a file storage device, a storage medium and an electronic device, which are applied to a file filing device, wherein the method comprises the following steps: acquiring a file group corresponding to each target storage medium in n target storage media, wherein the file group comprises a plurality of files stored in the target storage media, and each file group has the same data volume; acquiring a file list for recording the arrangement sequence of all files in the file group; generating m erasure correction files corresponding to the n file groups through a preset erasure correction algorithm; and writing each erasure correcting file in the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media. The multiple files stored in the storage medium can be used as independent computing units in an erasure correction algorithm to further generate corresponding erasure correction files, so that the step of segmenting or aggregating metadata of the files in an erasure correction file generation process is avoided, and the risks of data errors and loss in the erasure correction file generation and storage processes are reduced.
Description
Technical Field
The present disclosure relates to the field of information management, and in particular, to a file storage method, device, storage medium, and electronic device.
Background
Due to the popularization of digital information management, more and more companies adopt an automatic data backup and archiving system to perform file backup and archiving on data files related to company business. The data backup filing system relies on a file filing apparatus provided with a file writing device, a storage medium storage area, a file reading device, a database, and a processor, which requires a storage medium that uses optical storage media such as an optical disk and a magnetic tape, which have a longer life and a lower hardware annual failure rate, in addition to a magnetic disk. Based on the feature that the optical storage medium only allows one-time recording, in order to improve the reliability of system data and reduce the redundancy ratio of data storage copies, an EC (Erasure Code) file corresponding to a file is often generated while the file is archived. In this way, when an abnormality occurs in the optical storage medium, the document stored in the abnormal optical storage medium can be restored by erasing the document.
Disclosure of Invention
In order to overcome the problems in the related art, the present disclosure provides a file storage method, apparatus, storage medium, and electronic device.
According to a first aspect of the embodiments of the present disclosure, there is provided a file storage method applied to a file archiving device, the method including:
acquiring a first file group corresponding to each target storage medium in n target storage media, wherein the first file group comprises a plurality of files stored in the target storage media, and each first file group has the same data volume;
acquiring a file list corresponding to each first file group, wherein the file list is used for recording the arrangement sequence of all files in each first file group;
the method includes the steps that m erasure correction files corresponding to n first file groups are generated through a preset erasure correction algorithm, and the erasure correction proportion corresponding to the preset erasure correction algorithm is n: m;
and writing each erasure correcting file in the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media.
Optionally, each of the target storage media has the same storage capacity, and the obtaining a first file group corresponding to each of the n target storage media includes:
acquiring a second file group corresponding to each target storage medium, wherein the second file group consists of a plurality of files stored in the target storage medium;
when the data volume of the second file group is smaller than the storage capacity of the target storage medium, adding a preset identifier for the second file group to convert each second file group into a file group with a target data volume, wherein the target data volume is equal to the storage capacity of each target storage medium;
and taking the acquired n file groups with the target data volume as the n first file groups.
Optionally, the storage medium is an optical storage medium, and after writing each erasure correcting file of the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media, the method further includes:
and storing the n target storage media and the m erasure correcting file storage media through the file archiving device.
Optionally, the method further includes:
when an abnormal storage medium is detected, recovering a third file group stored in the abnormal storage medium through a target erasure file corresponding to the abnormal storage medium and a target file list; wherein the files in the third file group are arranged in the arrangement order recorded in the target file list.
Optionally, the abnormal storage medium is an abnormal storage medium detected in a process of reading a file, or an abnormal storage medium determined by a received abnormal repair instruction, and when the abnormal storage medium is detected, the file stored in the abnormal storage medium is recovered through an erasure correction file corresponding to the abnormal storage medium and a file list, including:
determining k target erasure correcting file storage media corresponding to the abnormal storage media from all erasure correcting file storage media stored in the file filing equipment;
acquiring k target erasure correcting files in the k target erasure correcting file storage media and the target file list;
acquiring a fourth file group according to the k target erasure correcting files;
extracting the third file group from the fourth file group according to the target file list, wherein the third file group consists of a plurality of files recorded in the target file list;
writing the third file group into an empty storage medium to replace the abnormal storage medium by the storage medium in which the third file group is stored.
According to a second aspect of the embodiments of the present disclosure, there is provided a file storage apparatus applied to a file filing device, the apparatus including:
a file group obtaining module, configured to obtain a first file group corresponding to each target storage medium in n target storage media, where the first file group includes multiple files stored in the target storage medium, and each first file group has the same data size;
a file list obtaining module, configured to obtain a file list corresponding to each first file group, where the file list is used to record an arrangement sequence of all files in each first file group;
the file generation module is configured to generate m erasure correction files corresponding to the n first file groups through a preset erasure correction algorithm, where an erasure correction ratio corresponding to the preset erasure correction algorithm is n: m;
and the file writing module is used for writing the file list and each erasure correcting file in m erasure correcting files corresponding to the n first file groups into an empty storage medium so as to obtain m erasure correcting file storage media.
Optionally, each of the target storage media has the same storage capacity, and the file group obtaining module is configured to:
acquiring a second file group corresponding to each target storage medium, wherein the second file group consists of a plurality of files stored in the target storage medium;
when the data volume of the second file group is smaller than the storage capacity of the target storage medium, adding a preset identifier for the second file group to convert each second file group into a file group with a target data volume, wherein the target data volume is equal to the storage capacity of each target storage medium;
and taking the acquired n file groups with the target data volume as the n first file groups.
Optionally, the storage medium is an optical storage medium, and the apparatus further includes:
and the storage medium storage module is used for storing the n target storage media and the m erasure correcting file storage media through the file filing equipment.
Optionally, the apparatus further comprises:
the file recovery module is used for recovering a third file group stored in the abnormal storage medium through a target erasure file corresponding to the abnormal storage medium and a target file list when the abnormal storage medium is detected; wherein the files in the third file group are arranged in the arrangement order recorded in the target file list.
Optionally, the abnormal storage medium is an abnormal storage medium detected in a process of reading a file, or an abnormal storage medium determined by a received abnormal repair instruction, and the file recovery module is configured to:
determining k target erasure correcting file storage media corresponding to the abnormal storage media from all erasure correcting file storage media stored in the file filing equipment;
acquiring k target erasure correcting files in the k target erasure correcting file storage media and the target file list;
acquiring a fourth file group according to the k target erasure correcting files;
extracting the third file group from the fourth file group according to the target file list, wherein the third file group consists of a plurality of files recorded in the target file list;
writing the third file group into an empty storage medium to replace the abnormal storage medium by the storage medium in which the third file group is stored.
According to a third aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the file storage method provided by the first aspect of the present disclosure.
According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the file storage method provided by the first aspect of the present disclosure.
By adopting the technical scheme provided by the disclosure, the following technical effects can be at least achieved:
the method comprises the steps that a first file group corresponding to each target storage medium in n target storage media can be obtained, the first file group comprises a plurality of files stored in the target storage media, and each first file group has the same data volume; acquiring a file list corresponding to each first file group, wherein the file list is used for recording the arrangement sequence of all files in each first file group; the n erasure correction files corresponding to the first file group are generated through a preset erasure correction algorithm, and the erasure correction proportion corresponding to the preset erasure correction algorithm is n: m; and writing each erasure correcting file in the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media. The multiple files stored in the storage medium can be used as independent computing units in the erasure correction algorithm to further generate corresponding erasure correction files, the step of segmenting or aggregating metadata in the files in the process of generating and storing the erasure correction files is avoided, and the risk of data errors and loss in the process of generating and storing the erasure correction files is reduced.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
FIG. 1 is a flow diagram illustrating a file storage method in accordance with an exemplary embodiment;
FIG. 2 is a flow chart of a method of writing a file according to the method shown in FIG. 1;
FIG. 3 is a flow chart of another file storage method according to that shown in FIG. 1;
FIG. 4 is a flow chart of yet another file storage method according to FIG. 3;
FIG. 5 is a flow chart of an exception file recovery method according to that shown in FIG. 4;
FIG. 6 is a block diagram illustrating a file storage device in accordance with an exemplary embodiment;
FIG. 7 is a block diagram illustrating an electronic device in accordance with an example embodiment.
Detailed Description
The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.
In the related art, a file to be archived is usually a packaged data file, and in an existing EC (Erasure Code) scheme, a ratio of an EC algorithm needs to be set to n: and m, wherein n is the number of original data files, and m is the number of erasure files. File storage schemes based on this EC scheme typically include: and dividing the packaged data file to be written into n parts, and calculating m parts of erasure correcting files according to the n parts of files. Then, the above n + m data files are recorded on different magnetic disks or optical disks respectively. The scheme needs to segment the whole data file, the storage is too scattered, and the risk of file data errors or loss in the file storage process is high.
Another file storage scheme based on the EC scheme is to parse the packaged data file into a plurality of metadata, and aggregate all the metadata into n files of a fixed size. And calculating m erasure correcting files through the n files, and further recording the n + m data files onto different magnetic disks or optical disks. However, the operation of re-aggregating after parsing the file changes the read-write flow of the original data file, and there is a risk of file data error or loss.
The inventor notices the problem and provides a new file storage method, which comprises the following specific steps:
before introducing the file storage method provided by the present disclosure, a target application scenario related to each embodiment in the present disclosure is first introduced, where the target application scenario includes a file archiving device, and the file archiving device includes: the device comprises a file writing device, a storage medium storage area, a file reading device, a database and a controller. The controller is used for controlling the operation of the file writing device, the storage medium storage area, the file reading device and the database.
FIG. 1 is a flowchart illustrating a file storage method applied to a file archiving device, according to an exemplary embodiment, the method including the steps of:
in step 101, a first file group corresponding to each of n target storage media is obtained.
The first file group comprises a plurality of files stored in the target storage medium, and each first file group has the same data size. The data size of the file group is the sum of the data sizes of all files in the file group.
For example, in the process of generating the Erasure correction file, the Erasure correction ratio of an EC (Erasure correction Code) algorithm (i.e. the following preset Erasure correction algorithm) is usually set to be n: and m, wherein n is the number of the original data files, and m is the number of the erasure files generated according to the n original data files. Before step 101, each file may be written into an empty storage medium in sequence according to an original arrangement order of a plurality of files to be archived, and when a storage medium is full, the storage medium is switched to a next empty storage medium until all files to be archived are written into the storage medium. When generating and storing the error correction file, it is first necessary to identify n storage media (i.e., the n target storage media) in which files have been written in the storage medium storage area. Then, the plurality of files stored in each target storage medium are used as a logically large file (i.e., a file group), and the sum of the data volumes of all files in each file group is made up to be the same as the storage capacity of the storage medium (assuming that the storage capacities of the n target storage media are the same), so that n first file groups can be obtained. It should be noted that the generation process of the first file group does not include the step of extracting, splitting or aggregating the files. In step 101, only a plurality of files stored in each target storage medium are recorded as a logical file group, and the logical file group is supplemented with data size to meet the requirement that the data size of n original data files in the EC scheme must be the same.
In step 102, a file list corresponding to each of the first file groups is obtained.
The file list is used for recording the arrangement sequence of all files in each first file group.
Illustratively, after the step 101, the files in each target storage medium correspond to a first file group containing a plurality of files, and the files in each first file group may be sequentially arranged in the target storage medium according to file names or file data amounts. In order to keep the file arrangement order in the target storage medium consistent with the file arrangement order in the original target storage medium when the files in the target storage medium are restored, the arrangement order of a plurality of files in each first file group needs to be acquired, and the file list needs to be generated. In addition, the file list can be used to record the file name and the data amount of each file in addition to the arrangement order of the files.
In step 103, m erasure correction files corresponding to the n first file groups are generated through a preset erasure correction algorithm.
Wherein, the erasure proportion corresponding to the preset erasure algorithm is n: and m is selected.
In step 104, each erasure correction file of the m erasure correction files and the file list are written to an empty storage medium to obtain m erasure correction file storage media.
For example, m erasure correction files corresponding to the n first file groups may be obtained according to an existing EC algorithm (i.e., the preset erasure correction algorithm), and then the m erasure correction files are written into m empty storage media by a file writing device in the file archiving device, respectively. The empty storage medium is a storage medium that does not store any files or data. In order to recover the abnormal storage media through the EC algorithm when the abnormal storage media occur in the n target storage media, the file list needs to be written into each of the m empty storage media. Each first file group corresponds to one file list, and the n first file groups correspond to the n file lists. The n file lists may be stored in each of the m empty storage media described above.
In summary, the technical solution provided by the embodiments of the present disclosure can obtain a first file group corresponding to each target storage medium in n target storage media, where the first file group includes a plurality of files stored in the target storage medium, and each first file group has the same data size; acquiring a file list corresponding to each first file group, wherein the file list is used for recording the arrangement sequence of all files in each first file group; the n erasure correction files corresponding to the first file group are generated through a preset erasure correction algorithm, and the erasure correction proportion corresponding to the preset erasure correction algorithm is n: m; and writing each erasure correcting file in the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media. The multiple files stored in the storage medium can be used as independent computing units in the erasure correction algorithm to further generate corresponding erasure correction files, the step of segmenting or aggregating metadata in the files in the process of generating and storing the erasure correction files is avoided, and the risk of data errors and loss in the process of generating and storing the erasure correction files is reduced.
Fig. 2 is a flowchart of a file writing method shown in fig. 1, where each target storage medium has the same storage capacity as shown in fig. 2, and the step 101 includes:
in step 1011, a second file group corresponding to each of the target storage media is obtained.
Wherein the second file group is composed of a plurality of files stored in the target storage medium.
For example, in the process of acquiring the first file group, a plurality of files stored in each target storage medium need to be acquired first, and then a file group is composed of the files as the second file group. Since the files stored in each target storage medium are different, the data size of the second file group is also different. The existing EC algorithm requires that the data volumes of n original data blocks are all the same, so that the calculation of the erasure file can be performed, and therefore, the data volume of the second file group needs to be supplemented.
In step 1012, when the data size of the second file group is smaller than the storage capacity of the target storage medium, a preset identifier is added to the second file group to convert each of the second file groups into a file group with the target data size.
Wherein the target data size is equal to the storage capacity of each of the target storage media.
In step 1013, the acquired n file groups having the target data amount are set as the n first file groups.
For example, the data amount of each second file group may be supplemented to the target data amount by adding a preset identifier, so that the following erasure correction file calculation step can be normally performed. Specifically, a plurality of meaningless preset marks (e.g., 0) may be added to the second file group described above to fill the data amount of each file group to the storage capacity of the target storage medium. As described above, the first file group is a logical file group, and the process of grouping files is actually to obtain a logically large file (i.e., the first file group) with the same data size, and use it as a calculation granularity in the EC algorithm. In the actual process of generating the erasure correction file, operations such as shifting, aggregation or conversion are not needed to be carried out on the file.
Fig. 3 is a flowchart of another file storage method shown in fig. 1, where the storage medium is an optical storage medium, and after step 104, the method may further include:
in step 105, the n target storage media and the m erasure correction file storage media are stored in the file archiving device.
Illustratively, the optical storage medium (e.g., blu-ray disc) has the property of supporting one-time data writing and multiple-time data reading, and is suitable for cold storage of data, so the optical storage medium is selected as the target storage medium and the erasure correction file storage medium in the embodiments of the present disclosure. And when the optical storage medium is a blue-ray disc, the disc cartridge is selected as a storage medium storage area of the file filing device. After the n target storage media and the m erasure correcting file storage media are obtained, the storage media may be stored in a storage medium storage area of the file filing apparatus.
Fig. 4 is a flowchart of still another file storage method shown in fig. 3, and as shown in fig. 4, after the step 105, the method may further include:
in step 106, when the abnormal storage medium is detected, the third file group stored in the abnormal storage medium is recovered through the target erasure file corresponding to the abnormal storage medium and the target file list.
Wherein the files in the third file group are arranged in the arrangement order recorded in the target file list.
Illustratively, the storage medium storage area of the file archiving device stores all storage media for storing original files and corresponding erasure-corrected file storage media. Due to inevitable reasons such as changes of storage environment and storage time, manual operation errors, quality defects of hardware equipment and the like, the files on the storage medium are damaged or lost and other abnormal conditions occur, and the files on the storage medium are unreadable. Wherein, the storage medium with the abnormality is the abnormal storage medium. When a storage medium which cannot be read is detected in the process of reading a file in the storage medium, or an abnormal repair instruction which is triggered by a worker and aims at a certain storage medium which is determined to be damaged is received, it can be determined that an abnormal storage medium is detected. At this time, all the files stored in the abnormal storage medium (i.e., the third file group) may be restored through the target erasure correction file and the target file list in the erasure correction file storage medium corresponding to the abnormal storage medium and other storage media (hereinafter, referred to as the same group storage medium) for storing the original file that are in the same group as the abnormal storage medium.
FIG. 5 is a flowchart of an abnormal file recovery method according to the method shown in FIG. 4, wherein the step 106, as shown in FIG. 5, includes:
in step 1061, k target erasure correction file storage media corresponding to the abnormal storage media are determined from all erasure correction file storage media stored in the file archiving device.
Illustratively, in generating an erasure correction file storage medium, the erasure correction ratio n in the EC algorithm: k, i.e., t storage media storing original files and k erasure-corrected file storage media as one EC group. For example, through the above steps 101-104, 100 storage media storing original files and 50 erasure-corrected file storage media are obtained, and it can be determined that the file archiving device includes 25 EC groups. When an abnormal storage medium is detected, an EC group where the abnormal storage medium is located needs to be determined, and erasure correcting file storage media in the EC group are k target erasure correcting file storage media corresponding to the abnormal storage medium. Taking an EC group including t storage media storing original files and k erasure correcting file storage media as an example, when it is determined that one of the storage media storing original files is an abnormal storage medium, the EC group may include k erasure correcting file storage media and t-1 other storage media storing original files as a basis for recovering the abnormal storage medium.
In step 1062, k target erasure correction files in the k target erasure correction file storage media and the target file list are obtained.
In step 1063, a fourth file group is obtained according to the k target erasure correction files.
For example, according to the k target erasure correction files, a fourth file group corresponding to the abnormal storage medium can be obtained from other storage media storing original files in the same EC group. In the embodiment of the present disclosure, since the erasure correction file is generated based on the logical large file (i.e., the first file group) synthesized from a plurality of original files, the logical large file (the fourth file group) corresponding to the abnormal storage medium is also restored when the file is restored. Further, in the following step 1064, the actual data files in the fourth file group need to be extracted according to the target file list, or it can be understood that the preset identifier added in the erasure correction file generation process in the fourth file group is removed, so as to obtain all the files stored in the abnormal storage medium, that is, all the files included in the third file group.
In step 1064, the third file group is extracted from the fourth file group according to the target file list.
Wherein the third file group is composed of a plurality of files recorded in the target file list.
For example, according to the k target erasure correction files, a fourth file group corresponding to the abnormal storage medium can be obtained from other storage media storing original files in the same EC group. In the embodiment of the present disclosure, since the erasure correction file is generated based on the logical large file (i.e., the first file group) synthesized from a plurality of original files, the logical large file (the fourth file group) corresponding to the abnormal storage medium is also restored when the file is restored. Further, in acquiring the fourth file group, in the step 1064, the actual data files in the fourth file group need to be extracted according to the target file list, or it can be understood that the preset identifier added in the erasure correction file generation process in the fourth file group is removed, so as to acquire all the files stored in the abnormal storage medium, that is, all the files included in the third file group. And then writing the third file group into a new storage medium and storing the third file group to finish the recovery of the abnormal storage medium.
In step 1065, the third file set is written to an empty storage medium to replace the abnormal storage medium with the storage medium storing the third file set.
In summary, the technical solution provided by the embodiments of the present disclosure can obtain a first file group corresponding to each target storage medium in n target storage media, where the first file group includes a plurality of files stored in the target storage medium, and each first file group has the same data size; acquiring a file list corresponding to each first file group, wherein the file list is used for recording the arrangement sequence of all files in each first file group; the n erasure correction files corresponding to the first file group are generated through a preset erasure correction algorithm, and the erasure correction proportion corresponding to the preset erasure correction algorithm is n: m; and writing each erasure correcting file in the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media. The method has the advantages that a plurality of files stored in the storage medium can be used as independent computing units in the erasure correction algorithm, so that corresponding erasure correction files are generated, the abnormal storage medium is recovered according to the generated erasure correction files and the file arrangement sequence when the storage medium is abnormal, the step of splitting or aggregating metadata in the files in the erasure correction file generation and storage process is avoided, the erasure correction file generation and abnormal data recovery process is realized under the condition that the file reading and writing flow is not changed, the risk of data abnormality in the erasure correction file generation and abnormal storage medium recovery process is reduced, and the stability of the file archiving system is improved.
Fig. 6 is a block diagram illustrating a file storage apparatus according to an exemplary embodiment, as shown in fig. 6, applied to a file filing device, the apparatus 600 including:
a file group obtaining module 610, configured to obtain a first file group corresponding to each target storage medium in n target storage media, where the first file group includes multiple files stored in the target storage medium, and each first file group has the same data size;
a file list obtaining module 620, configured to obtain a file list corresponding to each first file group, where the file list is used to record an arrangement sequence of all files in each first file group;
the file generating module 630 is configured to generate m erasure correction files corresponding to the n first file groups through a preset erasure correction algorithm, where an erasure correction ratio corresponding to the preset erasure correction algorithm is n: m;
a file writing module 640, configured to write the file list and each erasure correcting file in the m erasure correcting files corresponding to the n first file groups into an empty storage medium, so as to obtain m erasure correcting file storage media.
Optionally, each of the target storage media has the same storage capacity, and the file group acquiring module 610 is configured to:
acquiring a second file group corresponding to each target storage medium, wherein the second file group consists of a plurality of files stored in the target storage medium;
when the data size of the second file group is smaller than the storage capacity of the target storage medium, adding a preset identifier for the second file group to convert each second file group into a file group with a target data size, wherein the target data size is equal to the storage capacity of each target storage medium;
and taking the acquired n file groups with the target data size as the n first file groups.
Optionally, the storage medium is an optical storage medium, and the apparatus 600 further includes:
a storage medium storage module 650, configured to store the n target storage media and the m erasure-corrected file storage media by the file archiving device.
Optionally, the apparatus 600 further includes:
the file recovery module 660 is configured to, when an abnormal storage medium is detected, recover a third file group stored in the abnormal storage medium through a target erasure file corresponding to the abnormal storage medium and the target file list; wherein the files in the third file group are arranged in the arrangement order recorded in the target file list.
Optionally, the abnormal storage medium is an abnormal storage medium detected in a process of reading a file, or an abnormal storage medium determined by a received abnormal repair instruction, and the file recovery module 660 is configured to:
determining k target erasure correcting file storage media corresponding to the abnormal storage media from all erasure correcting file storage media stored in the file filing device;
acquiring k target erasure correcting files in the k target erasure correcting file storage media and the target file list;
acquiring a fourth file group according to the k target erasure correction files;
extracting the third file group from the fourth file group according to the target file list, wherein the third file group consists of a plurality of files recorded in the target file list;
and writing the third file group into an empty storage medium so as to replace the abnormal storage medium by the storage medium in which the third file group is stored.
In summary, the technical solution provided by the embodiments of the present disclosure can obtain a first file group corresponding to each target storage medium in n target storage media, where the first file group includes a plurality of files stored in the target storage medium, and each first file group has the same data size; acquiring a file list corresponding to each first file group, wherein the file list is used for recording the arrangement sequence of all files in each first file group; the n erasure correction files corresponding to the first file group are generated through a preset erasure correction algorithm, and the erasure correction proportion corresponding to the preset erasure correction algorithm is n: m; and writing each erasure correcting file in the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media. The method has the advantages that a plurality of files stored in the storage medium can be used as independent computing units in the erasure correction algorithm, so that corresponding erasure correction files are generated, the abnormal storage medium is recovered according to the generated erasure correction files and the file arrangement sequence when the storage medium is abnormal, the step of splitting or aggregating metadata in the files in the erasure correction file generation and storage processes is avoided, the erasure correction file generation and abnormal data recovery processes are realized under the condition that the file reading and writing processes are not changed, the risk of data abnormality in the erasure correction file generation and abnormal storage medium recovery processes is reduced, and the stability of the file archiving system is improved.
Illustratively, FIG. 7 is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. For example, electronic device 700 may be provided as a controller in a file archiving device. Referring to fig. 7, an electronic device 700 comprises a processor 701, which may be one or more in number, and a memory 702 for storing computer programs executable by the processor 701. The computer program stored in memory 702 may include one or more modules that each correspond to a set of instructions. Further, the processor 701 may be configured to execute the computer program to perform the file storage method described above.
Additionally, the electronic device 700 may also include a power component 703 and a communication component 704, the power component 703 may be configured to perform power management of the electronic device 700, and the communication component 704 may be configured to enable communication, e.g., wired or wireless communication, of the electronic device 700. The electronic device 700 may also include an input/output (I/O) interface 705. The electronic device 700 may operate based on an operating system stored in the memory 702, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, and so on.
In another exemplary embodiment, there is also provided a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the file storage method described above. For example, the computer readable storage medium may be the memory 702 described above including program instructions that are executable by the processor 701 of the electronic device 700 to perform the file storage method described above.
The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.
It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various combinations that are possible in the present disclosure are not described again.
In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.
Claims (8)
1. A method for storing files, applied to a file archiving apparatus, the method comprising:
acquiring a first file group corresponding to each target storage medium in n target storage media, wherein the first file group comprises a plurality of files stored in the target storage media, and each first file group has the same data volume;
acquiring a file list corresponding to each first file group, wherein the file list is used for recording the arrangement sequence of all files in each first file group;
the method includes the steps that m erasure correction files corresponding to n first file groups are generated through a preset erasure correction algorithm, and the erasure correction proportion corresponding to the preset erasure correction algorithm is n: m;
writing each erasure correcting file in the m erasure correcting files and the file list into an empty storage medium to obtain m erasure correcting file storage media;
each target storage medium has the same storage capacity, and the obtaining of the first file group corresponding to each target storage medium in the n target storage media includes:
acquiring a second file group corresponding to each target storage medium, wherein the second file group consists of a plurality of files stored in the target storage medium;
when the data volume of the second file group is smaller than the storage capacity of the target storage medium, adding a preset identifier for the second file group to convert each second file group into a file group with a target data volume, wherein the target data volume is equal to the storage capacity of each target storage medium;
and taking the acquired n file groups with the target data volume as the n first file groups.
2. The method of claim 1, wherein the storage medium is an optical storage medium, and after writing each of the m erasure-files and the list of files to an empty storage medium to obtain m erasure-file storage media, the method further comprises:
and storing the n target storage media and the m erasure correcting file storage media through the file archiving device.
3. The method of claim 1, further comprising:
when an abnormal storage medium is detected, recovering a third file group stored in the abnormal storage medium through a target erasure file corresponding to the abnormal storage medium and a target file list; wherein the files in the third file group are arranged in the arrangement order recorded in the target file list.
4. The method according to claim 3, wherein the abnormal storage medium is an abnormal storage medium detected in a process of reading a file or an abnormal storage medium determined by a received abnormal repair instruction, and when an abnormal storage medium is detected, restoring a file stored in the abnormal storage medium through an erasure file corresponding to the abnormal storage medium and a file list comprises:
determining k target erasure correcting file storage media corresponding to the abnormal storage media from all erasure correcting file storage media stored in the file filing equipment;
acquiring k target erasure correcting files in the k target erasure correcting file storage media and the target file list;
acquiring a fourth file group according to the k target erasure correcting files;
extracting the third file group from the fourth file group according to the target file list, wherein the third file group consists of a plurality of files recorded in the target file list;
writing the third file group into an empty storage medium to replace the abnormal storage medium by the storage medium in which the third file group is stored.
5. A file storage apparatus, for use in a file archiving device, the apparatus comprising:
a file group obtaining module, configured to obtain a first file group corresponding to each target storage medium in n target storage media, where the first file group includes multiple files stored in the target storage medium, and each first file group has the same data size;
a file list obtaining module, configured to obtain a file list corresponding to each first file group, where the file list is used to record an arrangement sequence of all files in each first file group;
the file generation module is configured to generate m erasure correction files corresponding to the n first file groups through a preset erasure correction algorithm, where an erasure correction ratio corresponding to the preset erasure correction algorithm is n: m;
a file writing module, configured to write the file list and each erasure correcting file of m erasure correcting files corresponding to the n first file groups into an empty storage medium, so as to obtain m erasure correcting file storage media;
each of the target storage media has the same storage capacity, and the file group acquisition module is configured to:
acquiring a second file group corresponding to each target storage medium, wherein the second file group consists of a plurality of files stored in the target storage medium;
when the data volume of the second file group is smaller than the storage capacity of the target storage medium, adding a preset identifier for the second file group to convert each second file group into a file group with a target data volume, wherein the target data volume is equal to the storage capacity of each target storage medium;
and taking the acquired n file groups with the target data volume as the n first file groups.
6. The apparatus of claim 5, wherein the storage medium is an optical storage medium, the apparatus further comprising:
and the storage medium storage module is used for storing the n target storage media and the m erasure correcting file storage media through the file filing equipment.
7. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 4.
8. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to carry out the steps of the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910563487.7A CN110347640B (en) | 2019-06-26 | 2019-06-26 | File storage method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910563487.7A CN110347640B (en) | 2019-06-26 | 2019-06-26 | File storage method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110347640A CN110347640A (en) | 2019-10-18 |
| CN110347640B true CN110347640B (en) | 2022-02-18 |
Family
ID=68183243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910563487.7A Active CN110347640B (en) | 2019-06-26 | 2019-06-26 | File storage method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110347640B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095013A (en) * | 2015-06-04 | 2015-11-25 | 华为技术有限公司 | Data storage method, data recovery method, and relevant device and system |
| CN106686095A (en) * | 2016-12-30 | 2017-05-17 | 郑州云海信息技术有限公司 | Data storage method and device based on erasure code technology |
| CN107608820A (en) * | 2017-09-26 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of file wiring method and relevant apparatus based on correcting and eleting codes |
| CN109213420A (en) * | 2017-06-29 | 2019-01-15 | 杭州海康威视数字技术股份有限公司 | Date storage method, apparatus and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8458287B2 (en) * | 2009-07-31 | 2013-06-04 | Microsoft Corporation | Erasure coded storage aggregation in data centers |
-
2019
- 2019-06-26 CN CN201910563487.7A patent/CN110347640B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095013A (en) * | 2015-06-04 | 2015-11-25 | 华为技术有限公司 | Data storage method, data recovery method, and relevant device and system |
| CN106686095A (en) * | 2016-12-30 | 2017-05-17 | 郑州云海信息技术有限公司 | Data storage method and device based on erasure code technology |
| CN109213420A (en) * | 2017-06-29 | 2019-01-15 | 杭州海康威视数字技术股份有限公司 | Date storage method, apparatus and system |
| CN107608820A (en) * | 2017-09-26 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of file wiring method and relevant apparatus based on correcting and eleting codes |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110347640A (en) | 2019-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9189311B2 (en) | Rebuilding a storage array | |
| US7133599B2 (en) | Realtime data recording method | |
| KR20180091989A (en) | Memory device having extending product lifetime and method of operating the memory device | |
| CN104811645A (en) | Embedded video-recording data storage method | |
| US8538925B2 (en) | System and method for backing up test data | |
| US20180357142A1 (en) | Reporting defects in a flash memory back-up system | |
| US9087014B1 (en) | Tape backup and restore in a disk storage environment | |
| US7234078B2 (en) | Data recovery method and data recording apparatus | |
| EP2669804A1 (en) | Storage system | |
| CN102222033B (en) | A kind of method and device for preserving small computer system interface access error | |
| CN110597779A (en) | Data reading and writing method in distributed file system and related device | |
| US10284230B2 (en) | Linked storage system and host system error correcting code | |
| CN111124740A (en) | Data reading method and device, storage equipment and machine-readable storage medium | |
| JP2015138372A (en) | Write inspection program, information processing apparatus, and method for write inspection | |
| CN103605587A (en) | Tape library data backup and filing method | |
| CN110347640B (en) | File storage method and device, storage medium and electronic equipment | |
| US11635914B2 (en) | Locating data within tape systems using sub dataset headers | |
| US20130212428A1 (en) | Information storage medium library array apparatus, information recording method and information recording program | |
| CN115509466B (en) | Data management method and device, electronic equipment and storage medium | |
| US7475278B2 (en) | Method, system and computer program product for recovery of formatting in repair of bad sectors in disk drives | |
| CN116431062A (en) | Data storage method, device, storage medium and system | |
| CN114064596A (en) | Log type embedded file system design method, system and storage medium | |
| US7623429B2 (en) | Method for the recovery of a damaged table of contents in an optical disc | |
| CN110389855B (en) | Magnetic tape library data verification method and device, electronic equipment and readable storage medium | |
| US20120262815A1 (en) | Method and system for dynamically expandable software based bad block management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221026 Address after: 100102 Room 01, Floor 3, Room 01, Building 2 to 4, Yard 6, Wangjing East Road, Chaoyang District, Beijing Patentee after: Beijing three cloud computing Co.,Ltd. Patentee after: BEIJING SANKUAI ONLINE TECHNOLOGY Co.,Ltd. Address before: 2106-030, No.9, Beisihuan West Road, Haidian District, Beijing 100190 Patentee before: BEIJING SANKUAI ONLINE TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |