CN111124307B - Data downloading and brushing method, device, equipment and readable storage medium - Google Patents
Data downloading and brushing method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN111124307B CN111124307B CN201911330457.8A CN201911330457A CN111124307B CN 111124307 B CN111124307 B CN 111124307B CN 201911330457 A CN201911330457 A CN 201911330457A CN 111124307 B CN111124307 B CN 111124307B
- Authority
- CN
- China
- Prior art keywords
- writing
- executing
- disk
- linked list
- fingerprint value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Abstract
The application discloses a data refreshing method, a device, equipment and a computer readable storage medium, wherein the method comprises the following steps: judging whether a fingerprint value identical to the fingerprint value exists in the metadata or not; if not, judging whether the IO which is the same as the fingerprint value of the IO and is executing disc writing exists, if so, storing the received IO in a linked list corresponding to the fingerprint value; adjusting the originally set timeout time corresponding to the IO of the executing disk writing according to the IO number in the linked list to obtain the adjusted timeout time; and judging whether the time for waiting for the flushing of the IO executing the disc writing reaches the corresponding adjusted overtime time or not, and if so, flushing the IO executing the disc writing. According to the technical scheme, the original setting timeout time corresponding to the IO of the disc being written is shortened through the IO number in the linked list, so that the efficiency of the disc brushing under the IO is improved, the processing efficiency of the IO in the linked list is improved, and the IO reading and writing efficiency of the host is improved.
Description
Technical Field
The present application relates to the field of full flash memory technologies, and more particularly, to a data flushing method, apparatus, device, and computer-readable storage medium.
Background
The full flash memory has the characteristics of high performance, flexibility and the like, is an important direction of memory development, and generally adopts an additional writing mode to write data, namely the data is written into the same logical address in a new place each time, and old data cannot be covered but is processed through garbage collection. In the process of processing IO output, because the full-flash storage is implemented by additionally writing and flashing the full stripe, a waiting time needs to be set so as to gather enough data to burst the full stripe, and meanwhile, in order to avoid that the full stripe cannot be burst for a long time, a timeout time needs to be added so that the data can be flashed even when the full stripe cannot be burst for a long time.
At present, for a scene with a high deduplication rate, because a large amount of issued IO data may have IO data with the same fingerprint value, for IO data with the same fingerprint value, it is necessary to wait for the first IO data to be written on a disk really completely, and then the remaining IO data that are the same as the fingerprint value and queued can be processed in sequence, so as to avoid the problems of repeatedly recording the fingerprint value (i.e., realizing deduplication) or returning the IO data without actually writing the disk. However, in the process of writing the first IO data on the disk, the newly received IO data having the same fingerprint value as the newly received IO data are queued to wait for the first IO data to complete writing the disk, and the first IO data can actually write the disk after the stripe where the first IO data is located meets the condition, so that a large amount of IO data can be queued and cannot be processed in time, and thus a large amount of IO data cannot be processed in time and a result cannot be returned to the host, and finally, the host reads and writes IO time delay occurs.
In summary, how to improve the efficiency of writing and reading IO data to a disk under IO data to improve the efficiency of writing and reading IO data by a host is a technical problem to be solved urgently by those skilled in the art.
Disclosure of Invention
In view of this, an object of the present application is to provide a data flushing method, apparatus, device and computer readable storage medium, for improving efficiency of flushing a disk under IO data, so as to improve efficiency of reading and writing IO of a host.
In order to achieve the above purpose, the present application provides the following technical solutions:
a data brushing method comprises the following steps:
receiving IO and calculating a fingerprint value of the IO, and judging whether a fingerprint value identical to the fingerprint value of the IO exists in metadata or not;
if the same fingerprint value does not exist, judging whether IO which is the same as the fingerprint value of the IO and is executing disc writing exists, if so, storing the received IO in a linked list corresponding to the fingerprint value of the IO executing disc writing;
when the IO number in the linked list reaches a preset value, adjusting the originally set timeout time corresponding to the IO of the executing disk writing according to the IO number in the linked list to obtain the adjusted timeout time; the adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a large IO number is shorter than the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a small IO number;
and judging whether the time for the IO executing the disc writing to wait for the flushing reaches the corresponding adjusted overtime time, and flushing the IO executing the disc writing if the time reaches the adjusted overtime time.
Preferably, when the number of the IO in the linked list is greater than a preset value, adjusting the originally set timeout time corresponding to the IO which is executing disk writing according to the number of the IO in the linked list, including:
setting priority for the IO executing disk writing according to the IO number in the linked list; the priority of the IO executing the disk writing corresponding to the linked list with the large IO quantity is higher than that of the IO executing the disk writing corresponding to the linked list with the small IO quantity;
judging whether the priority of the IO of the executing disk writing reaches a first threshold, if so, adjusting the originally set timeout time corresponding to the IO of the executing disk writing according to the priority of the IO of the executing disk writing; wherein the first threshold value corresponds to the preset value.
Preferably, before determining whether the time for the IO that is executing disc writing to wait for the flushing reaches the corresponding originally set timeout time, the method further includes:
judging whether the priority of the IO of the executing disc writing reaches a second threshold value, if so, flushing the IO of the executing disc writing; wherein the second threshold is greater than the first threshold.
Preferably, the determining whether the priority of the IO performing disk writing reaches a second threshold includes:
and judging whether the IO with the priority greater than the first threshold exists in the stripe where the IO which is executing disk writing is located, if so, determining that the IO which is executing disk writing reaches the second threshold.
Preferably, the determining whether the time for the IO that is executing disc writing to wait for the flushing reaches the corresponding adjusted timeout time includes:
judging whether the time of waiting for the flushing of the IO with the highest priority in the stripe where the IO executing the disk writing is located reaches the adjusted overtime corresponding to the IO with the highest priority, if so, determining that the time of waiting for flushing of the IO executing the disk writing reaches the adjusted overtime.
Preferably, after the flushing the IO that is writing to the disk, the method further includes:
and writing the fingerprint value of the IO of the executing disc writing into the metadata, and inserting a mapping from the virtual address to the physical address corresponding to the IO of the executing disc writing into the metadata.
Preferably, if there is a fingerprint value identical to the fingerprint value of the IO in the metadata, the method further includes:
inserting a mapping of virtual addresses to physical addresses corresponding to the IOs in the metadata.
A data swipe apparatus, comprising:
the receiving module is used for receiving IO and calculating a fingerprint value of the IO, and judging whether the metadata has the fingerprint value which is the same as the fingerprint value of the IO;
the first judgment module is used for judging whether the IO which is the same as the fingerprint value of the IO and is executing disc writing exists or not if the same fingerprint value does not exist, and storing the received IO in a linked list corresponding to the fingerprint value of the IO executing disc writing if the IO is the same as the fingerprint value of the IO and is executing disc writing;
the adjusting module is used for adjusting the originally set timeout time corresponding to the IO of the executing disk writing according to the IO number in the linked list so as to obtain the adjusted timeout time; the adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a large IO number is shorter than the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a small IO number;
and the brushing-down module is used for judging whether the time for waiting for brushing down of the IO executing the disc writing reaches the corresponding adjusted overtime time or not, and if so, brushing down the IO executing the disc writing.
A data swipe apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data brushing method as claimed in any one of the above when the computer program is executed.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of a data brushing method as claimed in any one of the preceding claims.
The application provides a data brushing method, a data brushing device, data brushing equipment and a computer readable storage medium, wherein the method comprises the following steps: receiving IO and calculating a fingerprint value of the IO, and judging whether the metadata has a fingerprint value which is the same as the fingerprint value of the IO; if the same fingerprint value does not exist, judging whether the fingerprint value of the IO is the same as that of the IO and the IO for writing the disk is executed, if so, storing the received IO in a linked list corresponding to the fingerprint value of the IO for writing the disk; when the number of IO in the linked list reaches a preset value, adjusting the originally set timeout time corresponding to the IO which is executing disk writing according to the number of IO in the linked list to obtain the adjusted timeout time; the adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a large IO number is smaller than the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a small IO number; and judging whether the time for waiting for the flushing of the IO executing the disc writing reaches the corresponding adjusted overtime time or not, and flushing the IO executing the disc writing if the time reaches the adjusted overtime time.
According to the technical scheme disclosed by the application, when the fact that the fingerprint value which is the same as the fingerprint value of the received IO does not exist in the metadata and the fact that the fingerprint value which is the same as the fingerprint value of the received IO and the IO for writing the disk is executed is judged to exist, the received IO is stored into the linked list corresponding to the fingerprint value of the IO for writing the disk, the originally set overtime time corresponding to the IO for writing the disk is reduced according to the number of the IOs in the linked list when the number of the IOs in the linked list reaches the preset value so as to obtain the adjusted overtime time, the IO for writing the disk can be brushed down when the time for waiting for brushing down by the IO for writing the disk reaches the corresponding adjusted overtime time, wherein the greater the number of the IOs in the linked list is, the shorter the adjusted overtime time of the corresponding IO for writing the disk is, so that the IO waiting time in the linked list corresponding to the IO can be processed as soon as possible, the original set timeout time corresponding to the IO of the writing disc being executed is shortened through the IO number in the linked list, the brushing time of the IO of the writing disc being executed is shortened, the efficiency of the IO of the writing disc being brushed is improved, the processing efficiency of the IO stored in the linked list is improved, and the IO reading and writing efficiency of the host is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data flushing method according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a data brushing apparatus according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a data brushing device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, which shows a flowchart of a data brushing method provided in an embodiment of the present application, a data brushing method provided in an embodiment of the present application may include:
s11: receiving IO and calculating a fingerprint value of the IO, and judging whether the metadata has a fingerprint value which is the same as the fingerprint value of the IO; if the same fingerprint value exists, the IO write-down disc is not executed, and if the same fingerprint value does not exist, step S12 is executed.
After the deduplication function is started, receiving IO, calculating a fingerprint value of the IO by using a hash algorithm or other algorithms, and judging whether a fingerprint value identical to the fingerprint value of the IO exists in metadata, if the same fingerprint value exists, it is indicated that the disk writing is realized by successful IO downspruing identical to the fingerprint value of the IO, at this time, deduplication can be performed, that is, the disk writing is not performed on the received IO, and if the same fingerprint value does not exist, it is indicated that the disk writing is realized by successful downspruing identical to the fingerprint value of the IO, at this time, it is possible to perform downspruing on the received IO for preparation.
S12: judging whether the IO which has the same fingerprint value as the IO and is executing disk writing exists; if there is no IO that is the same as the fingerprint value of the IO and is writing to the disk, the disk is flushed, and if there is an IO that is the same as the fingerprint value of the IO and is writing to the disk, step S13 is executed.
Before the received IO is subjected to disc flashing, whether the IO which has the same fingerprint value as the IO and is executing disc writing exists is judged, if not, the IO is used as the IO which is subjected to disc flashing first in the fingerprint values, and at the moment, the IO can be directly flashed.
S13: and storing the received IO in a linked list corresponding to the fingerprint value of the IO which is executing disk writing.
If the fingerprint value of the IO is the same as that of the IO and the IO writing to the disk is being executed, the received IO may be stored in the linked list corresponding to the fingerprint value of the IO writing to the disk, so that the IO stored in the linked list is processed after the IO writing to the disk is finished being flushed.
Each fingerprint value corresponds to a linked list, and the IOs stored in the linked list have the same fingerprint value.
S14: and when the IO number in the linked list reaches a preset value, adjusting the originally set timeout time corresponding to the IO which is executing disk writing according to the IO number in the linked list to obtain the adjusted timeout time.
The adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO writing to the disk corresponding to the chain table with a large number of IO is shorter than the adjusted timeout time of the IO writing to the disk corresponding to the chain table with a small number of IO.
Counting the number of IO in the linked list, and when the number of IO in the linked list reaches a preset value, adjusting the originally set timeout time corresponding to the IO executing the disk writing according to the number of IO in the linked list to obtain the adjusted timeout time, wherein the timeout time mentioned here specifically refers to that the disk writing can be performed when a strip where the IO executing the disk writing is located is not full but the time for waiting for the disk writing reaches the timeout time, and the originally set timeout time refers to the timeout time set when the IO disk writing is performed in the prior art.
In addition, the adjusted timeout time is shorter than the original set time, that is, the adjustment of the original set timeout time corresponding to the IO writing the disk mentioned herein specifically means to reduce the original set timeout time, so that the IO writing the disk can still be written down in a shorter time even though the IO writing the disk is not full of the tape, so as to shorten the time for writing the IO writing the disk, and improve the efficiency for writing the IO writing the disk, thereby facilitating the timely processing of the IO accumulated and queued in the linked list, so as to feed back the result to the host in time, and further improve the efficiency of the host in reading and writing the IO. In addition, when the originally set timeout time is adjusted, the larger the IO quantity in the linked list corresponding to the IO of the executing writing disk is, the larger the reduction degree of the originally set timeout time is, that is, the timeout time after the adjustment of the IO of the executing writing disk corresponding to the linked list with the large IO quantity is smaller than the timeout time corresponding to the linked list with the small IO quantity, so that the IO of the executing writing disk corresponding to the linked list with the large IO quantity can be efficiently and quickly refreshed, thereby the IO in the linked list corresponding to the IO can be quickly processed, the accumulated number of the IOs in the linked list can be reduced, the read-write IO efficiency of the host can be further improved, and the read-write IO delay can be reduced.
S15: and judging whether the time of the IO waiting for the flash-down of the disc writing reaches the corresponding adjusted timeout time, if so, executing the step S16, and if not, returning to the step S15.
After the originally set timeout is adjusted to obtain the adjusted timeout, when the stripe where the IO executing the disk writing is located is not full, it may be determined whether the time for the IO executing the disk writing to wait for the flushing reaches the adjusted timeout corresponding to the time, and if not, the step S15 may be returned to continue the determination when the stripe where the IO executing the disk writing is located is not full.
S16: the IO that is writing to the disk is being brushed down.
If the time for waiting for the flushing of the IO executing the disk writing is the adjusted timeout time when the stripe where the IO executing the disk writing is not full is not yet full, flushing the IO executing the disk writing, and processing the IO in the corresponding linked list after completing flushing the IO executing the disk writing.
According to the technical scheme disclosed by the application, when the fact that the fingerprint value which is the same as the fingerprint value of the received IO does not exist in the metadata and the fact that the fingerprint value which is the same as the fingerprint value of the received IO and the IO for writing the disk is executed is judged to exist, the received IO is stored into the linked list corresponding to the fingerprint value of the IO for writing the disk, the originally set overtime time corresponding to the IO for writing the disk is reduced according to the number of the IOs in the linked list when the number of the IOs in the linked list reaches the preset value so as to obtain the adjusted overtime time, the IO for writing the disk can be brushed down when the time for waiting for brushing down by the IO for writing the disk reaches the corresponding adjusted overtime time, wherein the greater the number of the IOs in the linked list is, the shorter the adjusted overtime time of the corresponding IO for writing the disk is, so that the IO waiting time in the linked list corresponding to the IO can be processed as soon as possible, the original set timeout time corresponding to the IO of the writing disc being executed is shortened through the IO number in the linked list, the brushing time of the IO of the writing disc being executed is shortened, the efficiency of the IO of the writing disc being brushed is improved, the processing efficiency of the IO stored in the linked list is improved, and the IO reading and writing efficiency of the host is improved.
According to the data flushing method provided by the embodiment of the application, when the number of IO in the linked list is greater than the preset value, the originally set timeout time corresponding to the IO executing the disk writing is adjusted according to the number of IO in the linked list, and the method may include:
setting priority for the IO executing disk writing according to the IO number in the linked list; the priority of the IO which is executing disk writing and corresponds to the linked list with the large IO quantity is higher than that of the IO which is executing disk writing and corresponds to the linked list with the small IO quantity;
judging whether the priority of the IO executing the disk writing reaches a first threshold, if so, adjusting the originally set timeout time corresponding to the IO executing the disk writing according to the priority of the IO executing the disk writing; wherein the first threshold corresponds to a preset value.
When the originally set timeout time corresponding to the IO which is executing disk writing is adjusted, priority may be set for the IO which is executing disk writing according to the number of the IO in the linked list, where the greater the number of the IO in the linked list, the higher the priority of the IO which is executing disk writing corresponding to the greater number of the IO in the linked list, that is, the higher the priority of the IO which is executing disk writing corresponding to the linked list with the greater number of the IO is than the priority of the IO which is executing disk writing corresponding to the linked list with the lesser number of the IO.
After the priority is set, whether the priority of the IO executing the disk writing reaches a first threshold corresponding to the preset value is judged, if yes, the original set timeout time corresponding to the IO executing the disk writing is shortened according to the priority of the IO executing the disk writing, and the adjusted timeout time shorter than the original set timeout time is obtained.
The mode of setting the priority for the IO executing the disk writing can be convenient for directly and clearly knowing which IO executing the disk writing needs to be flushed down preferentially, so as to shorten the corresponding original setting time.
The data flushing method provided in the embodiment of the present application may further include, before determining whether the time for waiting for flushing by an IO that is executing writing to a disk reaches the corresponding adjusted timeout time:
judging whether the priority of the IO executing the disk writing reaches a second threshold, if so, flushing the IO executing the disk writing; wherein the second threshold is greater than the first threshold.
Considering that a new IO may be stored in the linked list, therefore, the priority of the corresponding IO which is executing writing on the disk may change, in order to prevent the IO which is executing writing on the disk from being accumulated with more IO in the process of waiting for the flushing, and thus cannot be processed in time, before judging whether the time for waiting for flushing of the IO which is executing writing on the disk reaches the corresponding adjusted timeout time, it may be judged whether the priority of the IO which is executing writing on the disk reaches a second threshold (which is greater than the first threshold), if not, it may be judged whether the stripe where the IO is located is full, if not, it returns to step S15, and if so, the flushing is performed; if the priority of the IO of the disc being written reaches the second threshold, the IO of the disc being written is directly brushed down to shorten the brushing down time of the IO of the disc being written and improve the brushing down efficiency, so that the processing efficiency of the IO accumulated in the corresponding linked list is improved.
The data flushing method provided in the embodiment of the present application, which determines whether the priority of the IO that is executing disc writing reaches the second threshold, may include:
judging whether IO with the priority larger than a first threshold exists in a stripe where the IO which is executing disk writing is located, if so, determining that the IO which is executing disk writing reaches a second threshold.
When judging whether the priority of the IO performing disc writing reaches the second threshold, it may be judged whether an IO having a priority greater than the first threshold exists in a band where the IO performing disc writing is located (specifically, it may be judged by the priority of the IO performing disc writing having the highest priority), and if so, it is determined that the IO performing disc writing reaches the second threshold, at this time, it may be considered that all the IOs in the band reach the second threshold, and then, the band may be flushed down to improve flushing efficiency of the IO included in the band, thereby improving processing efficiency of the IO stored in a linked list corresponding to each IO in the band.
The data flushing method provided in the embodiment of the present application, which determines whether the time for waiting for flushing the IO that is executing writing to the disk reaches the corresponding adjusted timeout time, may include:
judging whether the time of waiting for the flushing of the IO with the highest priority in the strip where the IO which is executing the disk writing is located reaches the adjusted timeout time corresponding to the IO with the highest priority, and if so, determining that the time of waiting for the flushing of the IO which is executing the disk writing reaches the adjusted timeout time.
When judging whether the time for executing the IO wait for the flushing of the disc reaches the corresponding adjusted timeout time, it may be judged whether the time for executing the IO wait for the flushing with the highest priority in the band where the IO of the disc is located reaches the corresponding adjusted timeout time (the corresponding adjusted timeout time is the shortest in the band), and if so, it is determined that the IO executing the disc reaches the adjusted timeout time, at this time, it may be considered that all the IOs in the band reach the adjusted timeout time, and then the band may be flushed down to improve the flushing efficiency of the IO included in the band, thereby improving the processing efficiency of the IO stored in the linked list respectively corresponding to each IO in the band.
The data flushing method provided by the embodiment of the application, after flushing the IO that is writing a disk, may further include:
and writing the fingerprint value of the IO of the disc being written into the metadata, and inserting a mapping from the virtual address corresponding to the IO of the disc being written into the metadata to the physical address.
After the IO which is writing the disc is being brushed down, the fingerprint value of the IO which is writing the disc can be written into the metadata, so that subsequent IO which has the same fingerprint value as the IO can not be brushed down any more, and therefore deduplication is achieved.
In addition, a mapping from a virtual address corresponding to an IO that is writing to a disk may be inserted into the metadata, where the virtual address refers to an address written by the IO that is considered by a user to be writing to the disk, that is, an address operated by the user, and is a virtual address, and the physical address refers to an address of a location where the IO is actually written when the disk is flashed down, that is, an address of the corresponding flash disk, and is a real address, so that the IO can be directly obtained from the physical address through the mapping.
In the data flushing method provided in the embodiment of the present application, if a fingerprint value that is the same as the fingerprint value of the IO exists in the metadata, the method may further include:
a mapping of virtual addresses to physical addresses corresponding to IOs is inserted in the metadata.
If the metadata has a fingerprint value identical to that of the IO, at this time, the received IO does not write a disk down, but a mapping from a virtual address corresponding to the IO to a physical address may be inserted into the metadata, so that the corresponding data may be directly obtained from the physical address through the mapping.
The virtual address mentioned here refers to an address that a user thinks the IO is written into, and the physical address is an address of a position where a first IO having the same fingerprint value as the IO is actually written when a disk is written down, so that the IO having the same fingerprint value as the received IO can be obtained from the physical address through the mapping.
The embodiment of the present application further provides a data brushing device, refer to fig. 2, which shows a schematic structural diagram of the data brushing device provided in the embodiment of the present application, and the data brushing device may include:
a receiving module 21, configured to receive an IO and calculate a fingerprint value of the IO, and determine whether a fingerprint value identical to the fingerprint value of the IO exists in the metadata;
a first judging module 22, configured to, if there is no same fingerprint value, judge whether there is an IO that is the same as the fingerprint value of the IO and is executing disc writing, and if so, store the received IO in a linked list corresponding to the fingerprint value of the IO executing disc writing;
the adjusting module 23 is configured to adjust an originally set timeout time corresponding to the IO which is executing disk writing according to the number of the IO in the linked list, so as to obtain an adjusted timeout time; the adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO of the writing disk being executed corresponding to the linked list with a large IO number is shorter than the adjusted timeout time of the IO of the writing disk being executed corresponding to the linked list with a small IO number;
and the brushing-down module 24 is configured to determine whether a time for the IO executing the disc writing to wait for brushing down reaches a corresponding adjusted timeout time, and if so, brush down the IO executing the disc writing.
In the data brushing apparatus provided in the embodiment of the present application, the adjusting module 23 may include:
the setting unit is used for setting the priority of the IO which is executing disk writing according to the IO number in the linked list; the priority of the IO which is executing disk writing and corresponds to the linked list with the large IO quantity is higher than that of the IO which is executing disk writing and corresponds to the linked list with the small IO quantity;
the adjusting unit is used for judging whether the priority of the IO executing the disk writing reaches a first threshold value, if so, adjusting the adjusted timeout time corresponding to the IO executing the disk writing according to the priority of the IO executing the disk writing; wherein the first threshold corresponds to a preset value.
The data lower brushing device provided by the embodiment of the application can further comprise:
the second judging module is used for judging whether the priority of the IO executing the disc writing reaches a second threshold value or not before judging whether the time of the IO executing the disc writing waiting for the disc writing down reaches the corresponding adjusted overtime time or not, and if so, the IO executing the disc writing is down brushed; wherein the second threshold is greater than the first threshold.
The data brushing device provided by the embodiment of the application, the second judging module can comprise:
and the second judging unit is used for judging whether the IO with the priority greater than the first threshold exists in the stripe where the IO which is executing disk writing is located, and if so, determining that the IO which is executing disk writing reaches the second threshold.
In the data brushing apparatus provided in the embodiment of the present application, the first determining module 22 may include:
the first judging unit is configured to judge whether the time for the IO with the highest priority to wait for the flushing in the stripe where the IO writing the disk is currently executed reaches the adjusted timeout time corresponding to the IO with the highest priority, and if so, determine that the time for the IO writing the disk reaches the adjusted timeout time.
The data lower brushing device provided by the embodiment of the application can further comprise:
the first writing module is used for writing the fingerprint value of the IO of which the disc writing is being executed into the metadata after the IO of which the disc writing is being executed is flushed, and inserting a mapping from the virtual address corresponding to the IO of which the disc writing is being executed to the physical address into the metadata.
The data lower brushing device provided by the embodiment of the application can further comprise:
and a second writing module, configured to insert a mapping from a virtual address corresponding to the IO to a physical address in the metadata.
The embodiment of the present application further provides a data brushing device, see fig. 3, which shows a schematic structural diagram of the data brushing device provided in the embodiment of the present application, and the data brushing device may include:
a memory 31 for storing a computer program;
the processor 32, when executing the computer program stored in the memory 31, may implement the following steps:
receiving IO and calculating a fingerprint value of the IO, and judging whether the metadata has a fingerprint value which is the same as the fingerprint value of the IO; if the same fingerprint value does not exist, judging whether the fingerprint value of the IO is the same as that of the IO and the IO for writing the disk is executed, if so, storing the received IO in a linked list corresponding to the fingerprint value of the IO for writing the disk; when the number of IO in the linked list reaches a preset value, adjusting the originally set timeout time corresponding to the IO which is executing disk writing according to the number of IO in the linked list to obtain the adjusted timeout time; the adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO of the writing disk being executed corresponding to the linked list with a large IO number is shorter than the adjusted timeout time of the IO of the writing disk being executed corresponding to the linked list with a small IO number; and judging whether the time for the IO in which the disc writing is executed to wait for the flushing reaches the corresponding adjusted overtime time or not, and if so, flushing the IO in which the disc writing is executed.
The computer-readable storage medium provided in the embodiments of the present application stores a computer program thereon, and when executed by a processor, the computer program can implement the following steps:
receiving IO and calculating a fingerprint value of the IO, and judging whether the metadata has a fingerprint value which is the same as the fingerprint value of the IO; if the same fingerprint value does not exist, judging whether the fingerprint value of the IO is the same as that of the IO and the IO for writing the disk is executed, if so, storing the received IO in a linked list corresponding to the fingerprint value of the IO for writing the disk; when the number of IO in the linked list reaches a preset value, adjusting the originally set timeout time corresponding to the IO which is executing disk writing according to the number of IO in the linked list to obtain the adjusted timeout time; the adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a large IO number is smaller than the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a small IO number; and judging whether the time for waiting for the flushing of the IO executing the disc writing reaches the corresponding adjusted overtime time or not, and flushing the IO executing the disc writing if the time reaches the adjusted overtime time.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For a description of a relevant part in a data flushing device, equipment and a computer readable storage medium provided in the embodiments of the present application, reference may be made to the detailed description of a corresponding part in a data flushing method provided in the embodiments of the present application, and details are not described here again.
It should be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include elements inherent in the list. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. In addition, parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of corresponding technical solutions in the prior art, are not described in detail so as to avoid redundant description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (9)
1. A data brushing method is characterized by comprising the following steps:
receiving IO and calculating a fingerprint value of the IO, and judging whether a fingerprint value identical to the fingerprint value of the IO exists in metadata or not;
if the same fingerprint value does not exist, judging whether IO which is the same as the fingerprint value of the IO and is executing disc writing exists, if so, storing the received IO in a linked list corresponding to the fingerprint value of the IO executing disc writing; setting priority for the IO executing disk writing according to the IO number in the linked list; the priority of the IO which is executing disk writing and corresponds to the linked list with the large IO quantity is higher than that of the IO which is executing disk writing and corresponds to the linked list with the small IO quantity;
when the IO number in the linked list reaches a preset value, adjusting the originally set timeout time corresponding to the IO of the executing disk writing according to the IO number in the linked list to obtain the adjusted timeout time; the adjusted timeout time is shorter than the originally set timeout time, and the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a large IO number is shorter than the adjusted timeout time of the IO of the disk writing being executed corresponding to the linked list with a small IO number;
judging whether the time for the IO in which the disc writing is being executed to wait for the flushing reaches the corresponding adjusted timeout time, and flushing the IO in which the disc writing is being executed if the time reaches the adjusted timeout time;
judging whether the time of waiting for the flushing of the IO that is executing the disk writing reaches the corresponding adjusted timeout time or not, including:
judging whether the time of waiting for the flushing of the IO with the highest priority in the stripe where the IO executing the disk writing is located reaches the adjusted overtime corresponding to the IO with the highest priority, if so, determining that the time of waiting for flushing of the IO executing the disk writing reaches the adjusted overtime.
2. The data flushing method according to claim 1, wherein when the number of IO in the linked list is greater than a preset value, adjusting an originally set timeout time corresponding to the IO which is executing disc writing according to the number of IO in the linked list, includes:
judging whether the priority of the IO executing the disk writing reaches a first threshold value, if so, adjusting the originally set timeout time corresponding to the IO executing the disk writing according to the priority of the IO executing the disk writing; wherein the first threshold corresponds to the preset value.
3. The data brushing method according to claim 2, before determining whether the time for the IO that is writing a disc to wait for brushing reaches the corresponding adjusted timeout time, further comprising:
judging whether the priority of the IO of the executing disc writing reaches a second threshold value, if so, flushing the IO of the executing disc writing; wherein the second threshold is greater than the first threshold.
4. The data brushing method according to claim 3, wherein determining whether the priority of the IO that is performing disc writing reaches a second threshold comprises:
and judging whether the IO with the priority greater than the first threshold exists in the stripe where the IO which is executing disk writing is located, if so, determining that the IO which is executing disk writing reaches the second threshold.
5. The data brushing method according to claim 1, further comprising, after brushing the IO that is writing to the disk, the following:
and writing the fingerprint value of the IO of the executing disc writing into the metadata, and inserting a mapping from the virtual address to the physical address corresponding to the IO of the executing disc writing into the metadata.
6. The data brushing method according to claim 1, wherein if the same fingerprint value as the fingerprint value of the IO exists in the metadata, further comprising:
inserting a mapping of virtual addresses to physical addresses corresponding to the IOs in the metadata.
7. A data lower brushing device is characterized by comprising:
the receiving module is used for receiving IO and calculating a fingerprint value of the IO, and judging whether the metadata has the fingerprint value which is the same as the fingerprint value of the IO;
the first judgment module is used for judging whether the IO which is the same as the fingerprint value of the IO and is executing disc writing exists or not if the same fingerprint value does not exist, and storing the received IO in a linked list corresponding to the fingerprint value of the IO executing disc writing if the IO is the same as the fingerprint value of the IO and is executing disc writing; setting priority for the IO executing disk writing according to the IO number in the linked list; the priority of the IO which is executing disk writing and corresponds to the linked list with the large IO quantity is higher than that of the IO which is executing disk writing and corresponds to the linked list with the small IO quantity;
the adjusting module is used for adjusting the originally set timeout time corresponding to the IO of the executing disk writing according to the IO number in the linked list so as to obtain the adjusted timeout time; the timeout time after the adjustment of the IO of the writing disk is executed corresponding to the linked list with the large IO quantity is smaller than the timeout time after the adjustment of the IO of the writing disk is executed corresponding to the linked list with the small IO quantity;
the flushing module is used for judging whether the flushing waiting time of the IO which is executing the disc writing reaches the corresponding adjusted overtime time or not, and if so, flushing the IO which is executing the disc writing;
the lower brush module includes:
a first judging unit, configured to judge whether a time for the IO with the highest priority in a stripe where the IO performing disc writing is located to wait for the flushing reaches an adjusted timeout time corresponding to the IO with the highest priority, and if so, determine that the IO performing disc writing reaches the adjusted timeout time.
8. A data brushing apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data brushing method as claimed in any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data brushing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911330457.8A CN111124307B (en) | 2019-12-20 | 2019-12-20 | Data downloading and brushing method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911330457.8A CN111124307B (en) | 2019-12-20 | 2019-12-20 | Data downloading and brushing method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111124307A CN111124307A (en) | 2020-05-08 |
| CN111124307B true CN111124307B (en) | 2022-06-07 |
Family
ID=70501532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911330457.8A Active CN111124307B (en) | 2019-12-20 | 2019-12-20 | Data downloading and brushing method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111124307B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111857579B (en) * | 2020-06-30 | 2024-02-09 | 广东浪潮大数据研究有限公司 | SSD disk controller resetting method, SSD disk controller resetting system, SSD disk controller resetting device and readable storage medium |
| CN114489471B (en) * | 2021-08-10 | 2023-04-14 | 荣耀终端有限公司 | Input and output processing method and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017063495A1 (en) * | 2015-10-12 | 2017-04-20 | 中兴通讯股份有限公司 | Data migration method and apparatus |
| CN107944041A (en) * | 2017-12-14 | 2018-04-20 | 成都雅骏新能源汽车科技股份有限公司 | A kind of storage organization optimization method of HDFS |
| WO2018121455A1 (en) * | 2016-12-29 | 2018-07-05 | 华为技术有限公司 | Cached-data processing method and device, and storage controller |
| CN109460193A (en) * | 2018-11-15 | 2019-03-12 | 郑州云海信息技术有限公司 | I O process method, apparatus and terminal in a kind of storage system |
| CN110515557A (en) * | 2019-08-23 | 2019-11-29 | 北京浪潮数据技术有限公司 | A kind of cluster management method, device, equipment and readable storage medium storing program for executing |
-
2019
- 2019-12-20 CN CN201911330457.8A patent/CN111124307B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017063495A1 (en) * | 2015-10-12 | 2017-04-20 | 中兴通讯股份有限公司 | Data migration method and apparatus |
| WO2018121455A1 (en) * | 2016-12-29 | 2018-07-05 | 华为技术有限公司 | Cached-data processing method and device, and storage controller |
| CN107944041A (en) * | 2017-12-14 | 2018-04-20 | 成都雅骏新能源汽车科技股份有限公司 | A kind of storage organization optimization method of HDFS |
| CN109460193A (en) * | 2018-11-15 | 2019-03-12 | 郑州云海信息技术有限公司 | I O process method, apparatus and terminal in a kind of storage system |
| CN110515557A (en) * | 2019-08-23 | 2019-11-29 | 北京浪潮数据技术有限公司 | A kind of cluster management method, device, equipment and readable storage medium storing program for executing |
Non-Patent Citations (4)
| Title |
|---|
| B. Liu ; Y. S. Ma ; M. Shakezadeh ; H. L. Seet ; S. K. Yu."Maximum writing cycle control and disk lubricant and overcoat robustness of hybrid HAMR disk drives".《IEEE》.2013, * |
| 一种基于捎带回收的瓦记录Raid5写顺序化方法;张强等;《高技术通讯》;20180515(第05期);全文 * |
| 基于单向并行多链表的流管理;叶荻秋等;《计算机科学》;20130915(第09期);全文 * |
| 用户感知的重复数据删除算法;张沪寅等;《软件学报》;20151015(第10期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111124307A (en) | 2020-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220413706A1 (en) | Data Storage Method, Apparatus and Storage System | |
| US9141526B2 (en) | Reclaiming units by searching units for a predetermined criterion and storing data from a valid subunit | |
| TWI536163B (en) | Pre-emptive garbage collection of memory blocks | |
| JP5418808B2 (en) | Adaptive hybrid density memory storage control method and adaptive hybrid density memory storage | |
| KR101447188B1 (en) | Method and apparatus for controlling I/O to optimize flash memory | |
| CN106776095B (en) | Intelligent detection method and detection device for data reliability of SSD (solid State disk) | |
| CN111124307B (en) | Data downloading and brushing method, device, equipment and readable storage medium | |
| US7421624B2 (en) | Data recovery apparatus and method used for flash memory | |
| US11204864B2 (en) | Data storage devices and data processing methods for improving the accessing performance of the data storage devices | |
| CN111309687A (en) | Object storage small file processing method, device, equipment and storage medium | |
| US11176035B2 (en) | Data storage devices and data processing methods | |
| CN107168651B (en) | Small file aggregation storage processing method | |
| CN110377233B (en) | SSD (solid State disk) reading performance optimization method and device, computer equipment and storage medium | |
| US9666228B2 (en) | Shingled magnetic recording data store | |
| CN108763102B (en) | Garbage recycling method and device for solid state disk | |
| CN106293497B (en) | Watt record filesystem-aware in junk data recovery method and device | |
| CN114968834A (en) | Management mechanism for garbage recovery operation | |
| CN111177105B (en) | Mass file writing method, device, system and medium of distributed file system | |
| CN110597457A (en) | Solid state disk, control method of solid state disk and controller | |
| CN114237984A (en) | Recovery method and system of Trim data under abnormal power failure and solid state disk | |
| CN104794066A (en) | Storage apparatus and method for selecting storage area where data is written | |
| CN116185563B (en) | Software simulation algorithm based on vehicle-gauge microcontroller data flash memory | |
| CN112347001A (en) | Flash memory garbage recovery verification method and device and electronic equipment | |
| US10564886B2 (en) | Methods and apparatus for controlling flash translation layer recycle from host | |
| CN111158578B (en) | Storage space management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |