CN118012329A - File writing method and device, electronic equipment and storage medium - Google Patents
File writing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN118012329A CN118012329A CN202211402978.1A CN202211402978A CN118012329A CN 118012329 A CN118012329 A CN 118012329A CN 202211402978 A CN202211402978 A CN 202211402978A CN 118012329 A CN118012329 A CN 118012329A
- Authority
- CN
- China
- Prior art keywords
- current
- file
- stripe
- written
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000013467 fragmentation Methods 0.000 claims description 25
- 238000006062 fragmentation reaction Methods 0.000 claims description 25
- 239000012634 fragment Substances 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012163 sequencing technique Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 description 20
- 238000010586 diagram Methods 0.000 description 15
- 238000004364 calculation method Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 238000012937 correction Methods 0.000 description 5
- 239000002699 waste material Substances 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a file writing method, a file writing device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a current file to be written; if the current stripe is determined not to meet the writing condition of the current file to be written, adjusting the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold; and writing the current file to be written according to the current strip after the strip depth adjustment. By using the technical scheme of the invention, the situation that the file is stored to a plurality of nodes and a plurality of strips can be avoided, the file reading efficiency is improved, and the safety of the file storage is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for writing a file, an electronic device, and a storage medium.
Background
In the business of vehicle detection, face recognition, etc., small file storage of pictures, etc., is an important subject. To improve the security of file storage, erasure writing schemes after merging small files into large files are generally adopted.
In the erasure writing scheme, erasure coded data are distributed on different storage nodes, so that the data reliability of one node or a plurality of nodes under abnormal conditions can be ensured. The data organization of the erasure code can be expressed as n+m, N is the number of data fragments, M is the number of fragments of the check data, and the number of fragments of the check data means the maximum number of storage node downtime that can be tolerated. For example, when the data organization of the erasure code is 4+1, that is, 4 data fragments and 1 check data, it can tolerate that all data are still read normally after 1 storage device is down. However, in the existing file erasure writing mode, the stripe depth is fixed, which causes the situation that a small file is stored in a plurality of stripes of a plurality of nodes, so that a plurality of IOs are required for reading a complete file, and the file reading performance is affected.
Disclosure of Invention
The invention provides a file writing method, a file writing device, electronic equipment and a storage medium, which are used for improving file reading efficiency and file storage safety.
In a first aspect, an embodiment of the present invention provides a method for writing a file, where the method includes:
Acquiring a current file to be written;
If the current stripe is determined not to meet the writing condition of the current file to be written, adjusting the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold;
and writing the current file to be written according to the current strip after the strip depth adjustment.
In a second aspect, an embodiment of the present invention further provides a file writing device, where the device includes:
the current file to be written is obtained by the current file to be written obtaining module;
The strip depth adjusting module is used for adjusting the strip depth of the current strip according to the file size of the current file to be written and the strip depth threshold value if the current strip is determined not to meet the writing condition of the current file to be written;
and the current file to be written in writing module is used for writing the current file to be written in according to the current strip after the strip depth adjustment.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements a file writing method according to any one of the embodiments of the present invention when executing the program.
In a fourth aspect, embodiments of the present invention also provide a storage medium storing computer-executable instructions that, when executed by a computer processor, are configured to perform a file writing method according to any of the embodiments of the present invention.
According to the technical scheme, when the current stripe cannot realize the writing of the current file to be written, the stripe depth of the current stripe is adjusted according to the file size and the stripe depth threshold value of the current file to be written, and the writing of the current file to be written is performed according to the adjusted current stripe. According to the file size and the stripe depth threshold value of the current file to be written, the stripe depth of the current stripe is flexibly adjusted, the problems that in the prior art, the stripe depth is fixed, one file is stored in a plurality of stripes of a plurality of nodes, the complete file needs to be read for a plurality of times, the file reading performance is affected, the file reading efficiency is improved, and the safety of file storage is improved are solved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for writing a file according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of writing a file with a fixed stripe depth according to a first embodiment of the present invention;
FIG. 3 is a flowchart of a method for writing files according to a second embodiment of the present invention;
FIG. 4 is a schematic diagram of a first file writing process according to a second embodiment of the present invention;
FIG. 5 is a schematic diagram of a second file writing process according to a second embodiment of the present invention;
FIG. 6 is a schematic diagram of a third file writing process according to a second embodiment of the present invention;
FIG. 7 is a schematic diagram of a fourth file writing process according to a second embodiment of the present invention;
FIG. 8 is a schematic diagram of a file writing device according to a third embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a file writing method according to an embodiment of the present invention, where the method may be applied to a case of storing small files, and the method may be performed by a file writing device, where the file writing device may be implemented in a form of hardware and/or software, and the file writing device may be configured in an electronic device.
As shown in fig. 1, the method includes:
s110, acquiring a current file to be written.
It should be noted that, in this embodiment, the file to be written is a file with a smaller file size, and the criterion for determining whether erasure writing can be performed as the file to be written may be that the file size is smaller than or equal to a preset file capacity threshold. The file capacity threshold may be 512k, and the file to be written may be a picture, audio, or the like, but the type of the file to be written and the range of the file size are not limited in this embodiment.
The files to be written are typically consolidated prior to erasure writing of the files to be written. For example, taking n+m as 4+1, merging the first file, the second file, the third file and the fourth file into a large file, and writing the first file, the second file, the third file and the fourth file in sequence when writing the files.
And S120, if the current stripe is determined not to meet the writing condition of the current file to be written, adjusting the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold.
The current stripe is the first stripe which is not written with erasure correction and is selected in sequence according to the sequence of the stripes. One stripe spans all data storage nodes, with a portion of the stripe stored on each node. The size of the space of a stripe on a node is the stripe depth, and the number of nodes a stripe contains is called the stripe length.
The condition that the current file to be written is that the residual space of the current stripe on each node can support the current file to be written to write to two nodes at most. Correspondingly, the condition that the current file to be written is not met refers to the residual space of the current stripe on each node, the current file to be written is not supported to be written, or the current file to be written needs to be stored to three or more nodes.
The stripe depth threshold is an upper limit value of stripe depth, and may be 256k by way of example, but the value and the determination mode of the stripe depth threshold are not limited in this embodiment, and may be flexibly adjusted according to the actual application of writing a file.
FIG. 2 provides a schematic diagram of writing a file with a fixed stripe depth, as shown in FIG. 2, where the first file is divided into D1, D2, and D31, the second file is divided into D32, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13, and D141, the third file is divided into D142, D15, D16, D17, D18, and D191, and the fourth file is divided into D192 and D201, and D202 is a filled data block. It will be appreciated that the first file, although all in stripe0, is stored on three storage nodes, the second file is stored on stripe0-stripe3, and four nodes, and the third file is stored on stripe3, stripe4, and four nodes. This results in the first file, the second file, and the third file requiring multiple IOs to read the complete file when the file is read.
In the embodiment of the invention, the stripe depth of the current stripe is adjusted under the condition that the current stripe cannot support the current file to be written to write to at most two nodes. Compared with the mode that the stripe depth is fixed in the prior art, the method can ensure that the file to be written is stored on one stripe and spans two nodes at most, when the file is read, only the two nodes are required to be read, the situation that the same file to be written is stored on a plurality of nodes and a plurality of stripes is avoided, IO times when the file is read can be reduced, and the file reading efficiency is improved. Meanwhile, the same file to be written is guaranteed to be stored to two nodes at most, when a certain node is damaged, the probability that erasure recovery is needed to be carried out on the written file is reduced, the safety of data storage is improved, and the risk of data loss is also reduced.
Optionally, adjusting the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold may include: and if the strip depth threshold value is determined to be greater than or equal to one half of the file size of the current file to be written, adjusting the strip depth of the current strip according to one half of the file size of the current file to be written.
If the stripe depth threshold is greater than or equal to one-half the file size of the current file to be written, the stripe depth is adjusted to one-half the file size of the current file to be written. Specifically, one-half of the file size may be rounded up in 128k units. It can be appreciated that if the stripe depth threshold is smaller than one half of the file size of the current file to be written, even if the stripe depth is adjusted to the stripe depth threshold, the adjusted current stripe cannot support the current file to be written to at most two nodes, so that the stripe depth is not adjusted any more.
Optionally, if it is determined that the current stripe meets the writing condition of the current file to be written, writing the current file to be written into the remaining space of the current stripe on the target node. Wherein the number of target nodes is one or two.
In the embodiment of the invention, if the current stripe meets the writing condition of the current file to be written, the current file to be written is written into the residual space of the current stripe on one or two nodes. Specifically, the sorting can be performed according to the size of the remaining space of the current stripe on each node, one or two nodes with the size closest to the file size of the current file to be written are selected from each node, and the file writing is performed. Meanwhile, when the writing node is selected, the next node is selected after the residual space of the current stripe on one node is fully occupied as much as possible.
S130, writing the current file to be written according to the current stripe after the stripe depth adjustment.
Similarly, if the current stripe after the stripe depth adjustment meets the writing condition of the current file to be written, the current file to be written is written into the residual space of the current stripe on the target node.
In the embodiment of the invention, after the stripe depth is adjusted, if the adjusted current stripe can support the current file to be written to write to at most two nodes, the current file to be written to is written to the residual space of the current stripe on one or two nodes.
According to the technical scheme, when the current stripe cannot realize the writing of the current file to be written, the stripe depth of the current stripe is adjusted according to the file size and the stripe depth threshold value of the current file to be written, and the writing of the current file to be written is performed according to the adjusted current stripe. According to the file size and the stripe depth threshold value of the current file to be written, the stripe depth of the current stripe is flexibly adjusted, the problems that in the prior art, the stripe depth is fixed, one file is stored in a plurality of stripes of a plurality of nodes, the complete file needs to be read for a plurality of times, the file reading performance is affected, the file reading efficiency is improved, and the safety of file storage is improved are solved.
Example two
Fig. 3 is a flowchart of a file writing method provided by a second embodiment of the present invention, where based on the foregoing embodiment, a process of determining whether a current stripe meets a writing condition of a file to be written, a process of adjusting a stripe depth of the current stripe, and a process of writing the file to be written according to the adjusted current stripe are further specified.
As shown in fig. 3, the method includes:
S210, acquiring a current file to be written.
S220, sorting the residual spaces of the current strip on each node to obtain a first residual space and a second residual space.
And ordering the residual spaces of the current strip on each node according to the order from large to small, wherein the first residual space is the residual space of the first position of the space size rank, and the second residual space is the residual space of the second position of the space size rank.
It should be noted that, each node performing the remaining space sorting does not include an erasure code storage node, and the erasure code storage node is dedicated to storing erasure codes.
S230, judging and determining whether the sum of the size of the first residual space and the size of the second residual space is smaller than the file size of the current file to be written, if so, executing S240, otherwise, executing S260.
It can be understood that the meaning of comparing the sum of the size of the first remaining space and the size of the second remaining space with the file size of the current file to be written is that if the sum of the size of the first remaining space and the size of the second remaining space is smaller than the file size of the current file to be written, it indicates that the current file to be written still is written according to the current stripe depth, and it is impossible to store the current file to be written on only one or two nodes, and at this time, the stripe depth needs to be adjusted.
S240, judging whether the current stripe after stripe depth adjustment according to the file size and the stripe depth threshold of the current file to be written meets the writing condition of the current file to be written, if so, executing S250, otherwise, executing S270.
In the embodiment of the invention, if the writing condition of the current file to be written still cannot be met after the stripe depth adjustment is carried out according to the file size and the stripe depth threshold value of the current file to be written, the stripe depth adjustment is not carried out at this time, and the current file to be written is directly written into the next stripe.
S250, adjusting the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold.
In this embodiment, an initial value of the stripe depth may be predetermined, and an exemplary initial value of the stripe depth may be 128k, but the size and the determination manner of the initial value of the stripe depth are not limited in this embodiment. It will be appreciated that the stripe depth is this initial value prior to the stripe depth adjustment.
The specific process of adjusting the stripe depth is described in the above embodiments, and this embodiment is not described herein.
After the stripe depth is adjusted, if the current stripe supports that the current file to be written is stored in only one or two nodes, the current file to be written is stored in the remaining space of one or two nodes of the current stripe, and the process of determining the storage node is described in the above embodiment, which is not repeated herein.
S260, writing the current file to be written into the residual space of the current stripe on the target node.
The process of writing the current stripe to the current file to be written has been described in the above embodiment, and this embodiment is not described herein.
S270, writing the current file to be written according to the next stripe of the current stripe.
It will be appreciated that the current file to be written is written to the next stripe of the current stripe, still in the same manner as S220-S270, and the current file to be written is written when the next stripe supports that the current file to be written is stored on only one or two nodes.
S280, calculating the fragmentation rate of the current strip according to the size of the residual space of the current strip on each node and the size of the total space of the current strip.
Specifically, the sum of the sizes of the remaining spaces of the current stripe on each node may be calculated, and a value obtained by dividing the sum of the sizes of the remaining spaces by the size of the total space of the current stripe may be used as the fragmentation rate. The fragmentation rate is used for expressing the waste degree of the storage space of the current strip, and by calculating the fragmentation rate of the current strip and adopting different erasure writing modes of the current strip aiming at different fragmentation rates, the waste of the storage space of the strip can be reduced while ensuring that the files to be written are stored on two nodes of the same strip at most.
S290, according to the fragmentation rate of the current stripe, erasure writing of the current stripe is completed.
Accordingly, S290 may be implemented by the following steps A1-A2:
a1, if the current stripe erasure writing condition is met according to the fragmentation rate of the current stripe, filling the residual space of the current stripe on each node.
The remaining space of the current stripe on each node is filled, namely, the remaining space of the current stripe on each node is filled with 0.
Wherein, meeting the current stripe erasure writing condition may include: if the fragment rate of the current strip is determined to be greater than or equal to a preset fragment rate threshold value and is greater than or equal to a preset time threshold value, the file to be written is not stored, and the erasure writing condition of the current strip is determined to be met; or if the fragment rate of the current stripe is determined to be smaller than the preset fragment rate threshold, determining that the erasure writing condition of the current stripe is met.
Specifically, the preset fragmentation rate threshold may be determined by 0.5/m, where m represents the stripe length, but the determination manner of the preset fragmentation rate threshold is not limited in this embodiment.
It can be understood that if the fragmentation rate of the remaining space of the current stripe is smaller than the preset fragmentation rate threshold after the current file to be written is written in the next stripe, this indicates that the waste degree of the remaining space is within an acceptable range, and at this time, the remaining space of the current stripe on each node is directly filled with 0, and the erasure code is calculated and written.
If the fragment rate of the residual space of the current strip is greater than or equal to the preset fragment rate threshold, the waste degree of the residual space is larger, and the calculation of the erasure codes is suspended. Because the calculation and the writing of the erasure codes are not completed in the current stripe, the current stripe is not written, and in the subsequent writing process of the file to be written, the judgment of whether the current stripe meets the erasure writing condition of the current stripe can still be performed. That is, when the fragmentation rate of the remaining space is greater than or equal to a preset fragmentation rate threshold, the calculation of erasure codes is suspended, whether a file with a smaller size than the remaining space can be stored later or not is waited, and the files to be written are stored on at most two nodes. If the file to be written is written into the current stripe subsequently, so that the fragmentation rate of the current stripe is smaller than a preset fragmentation rate threshold, filling the residual space of the current stripe on each node to be 0, and performing erasure code calculation and writing. Or if the preset time threshold value is exceeded and a new file to be written is not stored in the current stripe, filling the residual space of the current stripe on each node to be 0, and performing erasure code calculation and writing.
A2, calculating erasure codes of the current stripe, and storing the erasure codes into an erasure code storage node to finish erasure writing of the current stripe.
The erasure code can be calculated by a Vandermonde matrix or a Reed-Solomon algorithm, etc., and the calculation mode of the erasure code is not limited in this embodiment. After the erasure correction code calculation is completed, the erasure correction code is stored to a position on the erasure correction code storage node, which is matched with the current stripe, so that erasure correction writing of the current stripe is completed.
In a specific example, taking the data organization of the erasure code as 4+1, merging the first file, the second file, the third file and the fourth file into a large file, and performing erasure writing as an example, and performing description of the file writing process. Fig. 4 provides a schematic diagram of a first file writing process, and in this example, taking an initial value of stripe depth of 128k as an example, the file size of the first file is 20k, and the current stripe is stripe0, which can be known that the current stripe meets the writing condition of the file to be written. And writing the first file into the storage node 1 corresponding to the strip 0.
FIG. 5 provides a schematic diagram of a second file writing process, as shown in FIG. 5, where the file size of the second file is 512k, and the stripe depth is required to be adjusted because one half of the file size is larger than the stripe depth of stripe 0. In this example, the stripe depth threshold is 256k, and one half of the file size is the same as the stripe depth threshold, so the stripe depth can be adjusted to 256k at this time. And writing the second file into the storage node 3 and the storage node 5 corresponding to the strip 0, wherein the storage node 3 and the storage node 5 corresponding to the strip 0 are occupied.
Fig. 6 provides a schematic diagram of a third file writing process, as shown in fig. 6, where the file size of the third file is 400k, because one half of the file size is smaller than the stripe depth of stripe0, where the stripe depth is kept constant by 256k, the remaining space of storage node 1 is selected in each storage node corresponding to stripe0, and storage node 6 performs writing of the third file. The third file is divided into 236k and 164k and stored in storage node 1 and storage node 6, respectively, and storage node 1 corresponding to stripe0 is fully occupied and storage node 6 is occupied 164k.
FIG. 7 provides a schematic diagram of a fourth file writing process, where the file size of the fourth file is 500k, and the remaining space of the current stripe on each storage node is only the remaining space of the storage node 6, and writing of the fourth file is not supported. Therefore, the fourth file is written in the next stripe1 at this time. Further, according to the initial stripe depth of stripe1 being 128k, the half of the file size of the fourth file being 200k, and the stripe depth threshold being 256k, the stripe depth of stripe1 may be adjusted to be one half of the file size of 200k, or the stripe depth of stripe1 may be directly adjusted to be 256k, in this embodiment, the description will be given taking the adjustment to be 256k as an example, the storage node 1 corresponding to stripe1 is fully occupied, and the storage node 3 is occupied with 244k.
Meanwhile, for strip 0, the remaining space only remains in the remaining space 92k of the storage node 6, and at this time, the obtained fragment rate value is compared with the preset fragment rate threshold value by dividing the remaining space 92k by the total space of strip 0. If the fragmentation rate is smaller than the preset fragmentation rate threshold, filling the remaining space of the storage node 6 with the strip 0, and calculating the erasure code to write the erasure code into the storage node 7 corresponding to the strip 0. If the fragmentation rate is greater than or equal to the preset fragmentation rate threshold, whether a file smaller than the remaining space is written in the subsequent process can be waited, if the new file is not written in the file which exceeds the preset time threshold, the remaining space of the storage node 6 is filled with the strip 0, and the storage node 7 corresponding to the erasure code written in the strip 0 is calculated.
According to the technical scheme, when the current stripe supports the writing of the current file to be written into at most two nodes, the current file to be written into one or two nodes of the current stripe is written into. When the current stripe can not realize the writing of the current file to be written, the stripe depth of the current stripe is adjusted according to the file size of the current file to be written and the stripe depth threshold value, and if the adjusted current stripe can not realize the writing of the current file to be written, the current file to be written is written to the next stripe. According to the file size and the stripe depth threshold value of the current file to be written, the stripe depth of the current stripe is flexibly adjusted, the problems that in the prior art, the stripe depth is fixed, one file is stored in a plurality of stripes of a plurality of nodes, the complete file needs to be read for a plurality of times, the file reading performance is affected, the file reading efficiency is improved, and the safety of file storage is improved are solved. Meanwhile, different erasure writing modes are selected for the current stripe according to different fragmentation rates, so that the waste of the residual space of the current stripe can be reduced.
Example III
Fig. 8 is a schematic structural diagram of a file writing device according to a third embodiment of the present invention. As shown in fig. 8, the apparatus includes: a current to-be-written file acquisition module 310, a stripe depth adjustment module 320, and a current to-be-written file writing module 330, wherein:
a current file to be written acquisition module 310, configured to acquire a current file to be written;
The stripe depth adjustment module 320 is configured to adjust the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold if it is determined that the current stripe does not meet the writing condition of the current file to be written;
The current to-be-written file writing module 330 is configured to write the current to-be-written file according to the current stripe after the stripe depth adjustment.
According to the technical scheme of the embodiment, when the current stripe cannot realize the writing of the current file to be written, the stripe depth of the current stripe is adjusted according to the file size and the stripe depth threshold value of the current file to be written, and the writing of the current file to be written is performed according to the adjusted current stripe. According to the file size and the stripe depth threshold value of the current file to be written, the stripe depth of the current stripe is flexibly adjusted, the problems that in the prior art, the stripe depth is fixed, one file is stored in a plurality of stripes of a plurality of nodes, the complete file needs to be read for a plurality of times, the file reading performance is affected, the file reading efficiency is improved, and the safety of file storage is improved are solved.
Based on the above embodiment, the stripe depth adjustment module 320 includes:
the residual space sequencing unit is used for sequencing the residual spaces of the current strip on each node to obtain a first residual space and a second residual space;
wherein each node for performing residual space sequencing does not comprise an erasure code storage node;
And the writing condition judging unit is used for determining that the current stripe does not meet the writing condition of the current file to be written if the sum of the size of the first residual space and the size of the second residual space is smaller than the file size of the current file to be written.
Based on the above embodiment, the stripe depth adjustment module 320 includes:
And the stripe depth adjusting unit is used for adjusting the stripe depth of the current stripe according to the half of the file size of the current file to be written if the stripe depth threshold value is determined to be larger than or equal to the half of the file size of the current file to be written.
On the basis of the above embodiment, the current file to be written writing module 330 includes:
The next band jumping unit is used for writing the current file to be written according to the next band of the current band if the current band after band depth adjustment according to the file size and the band depth threshold value of the current file to be written is determined and the writing condition of the current file to be written is not met;
The fragment rate calculation unit is used for calculating the fragment rate of the current strip according to the size of the residual space of the current strip on each node and the size of the total space of the current strip;
and the current stripe erasure writing unit is used for finishing erasure writing of the current stripe according to the fragmentation rate of the current stripe.
On the basis of the above embodiment, the present stripe erasure writing unit is specifically configured to:
if the current stripe erasure writing condition is met according to the fragment rate of the current stripe, filling the residual space of the current stripe on each node;
and calculating erasure codes for the current stripe, and storing the erasure codes into an erasure code storage node to finish erasure writing of the current stripe.
On the basis of the above embodiment, the present stripe erasure writing unit is specifically configured to:
If the fragment rate of the current strip is determined to be greater than or equal to a preset fragment rate threshold value and is greater than or equal to a preset time threshold value, the file to be written is not stored, and the erasure writing condition of the current strip is determined to be met;
Or if the fragment rate of the current stripe is determined to be smaller than the preset fragment rate threshold, determining that the erasure writing condition of the current stripe is met.
On the basis of the above embodiment, the apparatus further includes:
The target node writing module is used for writing the current to-be-written file into the residual space of the current stripe on the target node if the current stripe is determined to meet the writing condition of the current to-be-written file or if the current stripe with the depth adjusted is determined to meet the writing condition of the current to-be-written file;
Wherein the number of the target nodes is one or two.
The file writing device provided by the embodiment of the invention can execute the file writing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example IV
Fig. 9 shows a schematic diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 9, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (central processor), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the file writing method.
In some embodiments, the file writing method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the file writing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the file writing method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method of writing a file, comprising:
Acquiring a current file to be written;
If the current stripe is determined not to meet the writing condition of the current file to be written, adjusting the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold;
and writing the current file to be written according to the current strip after the strip depth adjustment.
2. The method of claim 1, wherein the current stripe does not satisfy a current file to be written writing condition, comprising:
Sequencing the residual spaces of the current strip on each node to obtain a first residual space and a second residual space;
wherein each node for performing residual space sequencing does not comprise an erasure code storage node;
If the sum of the size of the first residual space and the size of the second residual space is smaller than the file size of the current file to be written, determining that the current stripe does not meet the writing condition of the current file to be written.
3. The method of claim 1, wherein adjusting the stripe depth of the current stripe according to the file size of the current file to be written and the stripe depth threshold comprises:
And if the strip depth threshold value is determined to be greater than or equal to one half of the file size of the current file to be written, adjusting the strip depth of the current strip according to one half of the file size of the current file to be written.
4. The method of claim 1, further comprising, after determining that the current stripe does not satisfy the current file to be written writing condition:
If the current strip after the strip depth adjustment according to the file size and the strip depth threshold value of the current file to be written is determined, and the writing condition of the current file to be written is not met, writing the current file to be written according to the next strip of the current strip;
Calculating the fragmentation rate of the current strip according to the size of the residual space of the current strip on each node and the size of the total space of the current strip;
And according to the fragmentation rate of the current stripe, erasure writing of the current stripe is completed.
5. The method of claim 4, wherein completing erasure writing of the current stripe according to the fragmentation rate of the current stripe comprises:
if the current stripe erasure writing condition is met according to the fragment rate of the current stripe, filling the residual space of the current stripe on each node;
and calculating erasure codes for the current stripe, and storing the erasure codes into an erasure code storage node to finish erasure writing of the current stripe.
6. The method of claim 5, wherein determining that the current stripe erasure writing condition is satisfied based on the fragmentation rate of the current stripe comprises:
If the fragment rate of the current strip is determined to be greater than or equal to a preset fragment rate threshold value and is greater than or equal to a preset time threshold value, the file to be written is not stored, and the erasure writing condition of the current strip is determined to be met;
Or if the fragment rate of the current stripe is determined to be smaller than the preset fragment rate threshold, determining that the erasure writing condition of the current stripe is met.
7. The method according to claim 1 or 4, characterized in that the method further comprises:
If the current strip meets the writing condition of the current file to be written, or if the current strip with the depth adjusted strip meets the writing condition of the current file to be written, writing the current file to be written into the residual space of the current strip on the target node;
Wherein the number of the target nodes is one or two.
8. A document writing apparatus, comprising:
the current file to be written is obtained by the current file to be written obtaining module;
The strip depth adjusting module is used for adjusting the strip depth of the current strip according to the file size of the current file to be written and the strip depth threshold value if the current strip is determined not to meet the writing condition of the current file to be written;
and the current file to be written in writing module is used for writing the current file to be written in according to the current strip after the strip depth adjustment.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the file writing method according to any of claims 1-7 when executing the program.
10. A storage medium storing computer executable instructions which, when executed by a computer processor, are for performing the file writing method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211402978.1A CN118012329A (en) | 2022-11-09 | 2022-11-09 | File writing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211402978.1A CN118012329A (en) | 2022-11-09 | 2022-11-09 | File writing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118012329A true CN118012329A (en) | 2024-05-10 |
Family
ID=90956874
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211402978.1A Pending CN118012329A (en) | 2022-11-09 | 2022-11-09 | File writing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118012329A (en) |
-
2022
- 2022-11-09 CN CN202211402978.1A patent/CN118012329A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112988727B (en) | Data annotation method, device, equipment, storage medium and computer program product | |
| CN112540731B (en) | Data append writing method, device, equipment, medium and program product | |
| CN114996173B (en) | Method and device for managing write operation of storage equipment | |
| CN116451174A (en) | Task execution device, method, electronic device, and storage medium | |
| CN112650449B (en) | Method and system for releasing cache space, electronic device and storage medium | |
| CN118012329A (en) | File writing method and device, electronic equipment and storage medium | |
| CN114564149B (en) | Data storage method, device, equipment and storage medium | |
| CN115617800A (en) | Data reading method and device, electronic equipment and storage medium | |
| CN115774602A (en) | Container resource allocation method, device, equipment and storage medium | |
| CN115438007A (en) | File merging method and device, electronic equipment and medium | |
| CN115564329A (en) | Typical capacity scene determining method, device, equipment and storage medium | |
| CN113657408A (en) | Method and device for determining image characteristics, electronic equipment and storage medium | |
| CN114999665A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114879905A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN113327194A (en) | Image style migration method, device, equipment and storage medium | |
| CN112396100A (en) | Fine-grained classification model optimization method, system and related device | |
| CN113326890B (en) | Labeling data processing method, related device and computer program product | |
| CN116719621B (en) | Data write-back method, device, equipment and medium for mass tasks | |
| CN113326888B (en) | Labeling capability information determining method, related device and computer program product | |
| CN116662061A (en) | Data storage method, device, equipment and medium of capacity-expansion erasure code storage system | |
| CN115840645A (en) | Method, device, storage medium and electronic equipment for predicting load index | |
| CN116801001A (en) | Video stream processing method and device, electronic equipment and storage medium | |
| CN118132799A (en) | Video stream storage method, device, network video recording equipment and medium | |
| CN113965206A (en) | Compression method, compression device, electronic equipment and storage medium | |
| CN117633550A (en) | Blocking mode matching method and medium based on FM-index algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |