CN115113824A - Data object brushing method, device, equipment and medium - Google Patents
Data object brushing method, device, equipment and medium Download PDFInfo
- Publication number
- CN115113824A CN115113824A CN202210868345.3A CN202210868345A CN115113824A CN 115113824 A CN115113824 A CN 115113824A CN 202210868345 A CN202210868345 A CN 202210868345A CN 115113824 A CN115113824 A CN 115113824A
- Authority
- CN
- China
- Prior art keywords
- dirty data
- data object
- dirty
- preset
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a data object brushing method, a device, equipment and a medium, which relate to the technical field of computers and comprise the following steps: determining the modified data objects in the solid state disk as dirty data objects, and adding the dirty data objects to a dirty data object queue; dividing the dirty data object according to the preset unit length of the mechanical hard disk to obtain a plurality of division areas; the dirty data in the division areas with the total dirty data amount larger than the lower brushing threshold value are brushed down to the mechanical hard disk; and adding the flushed dirty data object into the target queue, judging whether dirty data exist in the flushed dirty data object, if so, adding the dirty data object into the dirty data object queue again, and if not, releasing the flushed data object in the target queue. According to the method and the device, the dirty data in the division area with the total dirty data amount larger than the lower brushing threshold value are preferably brushed down, the condition that the dirty data are brushed down and scattered is avoided, therefore, the lower brushing aggregation degree is increased, and the pressure on the mechanical hard disk is reduced.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for flushing a data object.
Background
SSD (Solid State Drives, Solid State Disk) has higher random IO (Input/Output) performance, so that many current storage systems use SSD as a cache Disk, service IO is written into the cache Disk first, and then data is brushed from the SSD cache Disk to the HDD (Hard Disk Drive) by asynchronous brushing. Currently, most storage systems use data objects as basic units for processing data of the system, so that the entire objects are usually used as minimum units for refreshing. After a certain data object is modified by a service IO, the data object is marked as a dirty data object and is added into a dirty data object LRU (Least Recently Used) queue, then a flushing module takes out the dirty data object from the dirty data object LRU, calculates a dirty data segment of the dirty data object and flushes the dirty data object, and after the flushing is finished, the data object is added into a target data object LRU queue to indicate that all data of the object are written into a main memory but the data in a cache are not deleted yet, and the target data of the object needs to be released when a subsequent cache space is recycled.
However, a data object is usually 4M or larger in size, and in a random small IO service scenario, hundreds of dirty data segments may be generated on a data object, and among the dirty data segments, some data segments may be aggregated into a larger dirty data segment, but other dirty data segments are still dispersed. Therefore, when the whole data object is flushed, the HDD needs to be frequently tracked by a large number of random IO, a large pressure is instantaneously applied to the HDD, the flushing efficiency is low, and the service performance is affected by the increase of the time delay caused by the busy disk.
Therefore, how to avoid refreshing scattered dirty data and reduce the number of seek times of the mechanical hard disk to reduce the pressure on the mechanical hard disk and improve the service performance is an urgent problem to be solved in the field.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a method, an apparatus, a device, and a medium for flushing a data object, which can avoid flushing dirty data and reduce the number of seeks of a mechanical hard disk, so as to reduce the pressure on the mechanical hard disk and improve the service performance, and the specific scheme thereof is as follows:
in a first aspect, the present application discloses a data object brushing method, comprising:
when detecting that a client modifies a data object in a solid state disk, determining that the data object is a dirty data object, and adding the dirty data object to a preset dirty data object queue;
determining the dirty data objects from the dirty data object queue, and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions;
judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is larger than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data larger than the preset dirty data amount lower brushing threshold value to the mechanical hard disk;
adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue.
Optionally, when a modification operation of a client on a data object in a solid state disk is detected, determining that the data object is a dirty data object, and adding the dirty data object to a preset dirty data object queue, where the method includes:
when detecting that a client modifies a data segment in a data object in a solid state disk, determining that the data object is a dirty data object, determining that the data segment is a dirty data segment, and adding the dirty data object to a preset dirty data object queue.
Optionally, before determining whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition is greater than a preset dirty data amount lower brushing threshold, the method further includes:
determining the total amount of dirty data and the number of dirty data objects in the dirty data object queue, and determining the quotient of the total amount of dirty data in the dirty data object queue and the number of dirty data objects in the dirty data object queue as a first parameter;
determining the size of a single dirty data object in the dirty data object queue, and determining the quotient of the size of the single dirty data object in the dirty data object queue and the preset unit length as a second parameter;
determining an average dirty data amount corresponding to the divided regions based on a quotient of the first parameter and the second parameter, and determining the average dirty data amount as the preset dirty data amount lower brushing threshold.
Optionally, the data object brushing method further includes:
and calculating the average dirty data volume according to a preset time interval, and updating the preset dirty data volume lower brushing threshold value by using the average dirty data volume.
Optionally, after the dirty data in the partition area whose total amount of dirty data is greater than the preset dirty data amount lower brushing threshold is flushed to the mechanical hard disk, the method further includes:
and marking a dirty data segment corresponding to the dirty data which is flushed to the mechanical hard disk as a target data segment.
Optionally, the performing data release on the flushed data object in the target object queue includes:
and releasing the data of the target data segment in the data object after being brushed down in the target object queue.
Optionally, after determining whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition is greater than a preset dirty data amount lower brushing threshold, the method further includes:
and if the total amount of dirty data of each data segment in the dirty data object corresponding to the partition interval is not greater than the preset dirty data amount lower brushing threshold value, re-executing the step of adding the dirty data object to a preset dirty data object queue.
In a second aspect, the present application discloses a data object brushing apparatus, comprising:
the dirty data object adding module is used for determining that the data object is a dirty data object when the modification operation of the client on the data object in the solid state disk is detected, and adding the dirty data object to a preset dirty data object queue;
the dirty data object dividing module is used for determining the dirty data objects from the dirty data object queue and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions;
the dirty data brushing module is used for judging whether the total dirty data amount of each data segment in the dirty data object corresponding to each partition interval is larger than a preset dirty data amount brushing threshold value or not, and brushing the dirty data in the partition interval with the total dirty data amount larger than the preset dirty data amount brushing threshold value to the mechanical hard disk;
and the data release module is used for adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the data object brushing method disclosed above.
In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements the data object brushing method disclosed above.
The method comprises the steps that when modification operation of a client on a data object in a solid state disk is detected, the data object is determined to be a dirty data object, and the dirty data object is added to a preset dirty data object queue; determining the dirty data objects from the dirty data object queue, and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions; judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is larger than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data larger than the preset dirty data amount lower brushing threshold value to the mechanical hard disk; adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue. Therefore, the dirty data objects are divided according to the preset unit length of the mechanical hard disk to obtain a plurality of division areas, the dirty data in the division areas with the total dirty data amount larger than the dirty data amount lower brushing threshold value are preferably brushed, and therefore the lower brushing aggregation degree is increased and the mechanical hard disk seek times are reduced by avoiding brushing the lower scattered dirty data, and further the pressure on the mechanical hard disk is reduced and the service performance is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a data object brushing method disclosed herein;
FIG. 2 is a flow chart of a particular data object rendering method disclosed herein;
FIG. 3 is a schematic diagram of a data object brushing method disclosed in the present application;
FIG. 4 is a flow chart of a particular data object rendering method disclosed herein;
FIG. 5 is a schematic diagram of a data object brushing apparatus according to the present disclosure;
fig. 6 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When the whole data object is flushed, the HDD needs to be frequently tracked by a large number of random IO, a large pressure is instantaneously applied to the HDD, the flushing efficiency is low, and the service performance is affected by the increase of the time delay due to the busy disk.
Therefore, the embodiment of the application provides a data object flushing scheme, which can avoid flushing of scattered dirty data and reduce the number of times of track seeking of a mechanical hard disk, so as to reduce the pressure on the mechanical hard disk and improve the service performance.
The embodiment of the application discloses a data object brushing method, and as shown in figure 1, the method comprises the following steps:
step S11: when the modification operation of a client on a data object in a solid state disk is detected, determining that the data object is a dirty data object, and adding the dirty data object to a preset dirty data object queue.
In a specific implementation manner, when a modification operation of a client on a data object in a solid state disk is detected, determining that the data object is a dirty data object, and adding the dirty data object to a preset dirty data object queue;
in another specific implementation manner, when a new write operation of a client on a data object in a solid state disk is detected, the data object is determined to be a dirty data object, and the dirty data object is added to a preset dirty data object queue.
Step S12: and determining the dirty data object from the dirty data object queue, and dividing the dirty data object according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions.
In this embodiment, the preset unit length may be the minimum allocation unit length of the mechanical hard disk, for example, 256K, that is, if the dirty data object is 4M, 0K to 256K are determined as the first division, 256K to 512K are determined as the second division, and so on. Because the storage system distributes the actual physical space according to the minimum distribution unit when writing data on the HDD, the data objects are divided according to the minimum distribution unit of the HDD, so that the data objects in the same minimum distribution unit can be ensured to be continuous on the HDD, and the seek time of the HDD can not be increased even if the data objects are discontinuous to a certain extent during the brushing-down, thereby not causing excessive pressure on the HDD.
Step S13: and judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is greater than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data greater than the preset dirty data amount lower brushing threshold value to the mechanical hard disk.
In this embodiment, if the total amount of dirty data of each data segment in the dirty data object corresponding to the partition is not greater than the preset dirty data amount lower brushing threshold, the step of adding the dirty data object to the preset dirty data object queue is executed again.
It should be noted that the process of determining the preset dirty data amount lower brushing threshold specifically includes: determining the total amount of dirty data and the number of dirty data objects in the dirty data object queue, and determining the quotient of the total amount of dirty data in the dirty data object queue and the number of dirty data objects in the dirty data object queue as a first parameter; determining the size of a single dirty data object in the dirty data object queue, and determining the quotient of the size of the single dirty data object in the dirty data object queue and the preset unit length as a second parameter; determining an average dirty data amount corresponding to the divided regions based on a quotient of the first parameter and the second parameter, and determining the average dirty data amount as the preset dirty data amount lower brushing threshold. And calculating the average dirty data amount according to a preset time interval, and updating the preset dirty data amount lower brushing threshold by using the average dirty data amount, so that the preset dirty data amount lower brushing threshold is updated in real time according to the current dirty data statistical condition, and therefore, the condition that dirty data which can be brushed down cannot be found is avoided.
In this embodiment, the dirty data in the division areas, of which the total dirty data amount is greater than the preset dirty data amount lower brushing threshold, are brushed down to the mechanical hard disk, so that the lower brushing aggregation degree is increased and the mechanical hard disk seek times are reduced by avoiding brushing down the scattered dirty data, and further the pressure on the mechanical hard disk is reduced and the service performance is improved.
Step S14: adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue.
In this embodiment, the dirty data object after being flushed is added to a preset target object queue, and it is determined whether dirty data exists in the dirty data object after being flushed, if so, the dirty data object is added to the preset dirty data object queue again, and if not, the data object after being flushed in the target object queue is released.
The method comprises the steps that when modification operation of a client on a data object in a solid state disk is detected, the data object is determined to be a dirty data object, and the dirty data object is added to a preset dirty data object queue; determining the dirty data objects from the dirty data object queue, and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions; judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is larger than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data larger than the preset dirty data amount lower brushing threshold value to the mechanical hard disk; adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue. Therefore, the dirty data objects are divided according to the preset unit length of the mechanical hard disk to obtain a plurality of division areas, the dirty data in the division areas with the total dirty data amount larger than the dirty data amount lower brushing threshold value are preferably brushed, and therefore the lower brushing aggregation degree is increased and the mechanical hard disk seek times are reduced by avoiding brushing the lower scattered dirty data, and further the pressure on the mechanical hard disk is reduced and the service performance is improved.
The embodiment of the application discloses a specific data object brushing method, and compared with the previous embodiment, the embodiment further describes and optimizes the technical scheme. Referring to fig. 2, the method specifically includes:
step S21: when detecting that a client modifies a data segment in a data object in a solid state disk, determining that the data object is a dirty data object, determining that the data segment is a dirty data segment, and adding the dirty data object to a preset dirty data object queue.
In this embodiment, the modified data segment in the data object is determined as the dirty data segment.
Step S22: and determining the dirty data object from the dirty data object queue, and dividing the dirty data object according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions.
Step S23: and judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is greater than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data greater than the preset dirty data amount lower brushing threshold value to the mechanical hard disk.
Step S24: and marking a dirty data segment corresponding to the dirty data which is flushed to the mechanical hard disk as a target data segment.
In this embodiment, after the dirty data in the division areas whose total amount of dirty data is greater than the preset dirty data amount lower brushing threshold is flushed to the mechanical hard disk, the dirty data segment corresponding to the dirty data that has been flushed to the mechanical hard disk is marked as the target data segment.
Step S25: adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing data of the target data segment in the flushed data object in the target object queue.
In this embodiment, the flushed dirty data object is added to a preset target object queue, and it is determined whether dirty data exists in the flushed dirty data object, and if not, data release is performed on the target data segment in the flushed data object in the target object queue, so that a cache space is released, and pressure of a solid state disk is reduced.
For more specific working processes of step S22 and step S23, reference is made to the foregoing embodiments, and details are not repeated here.
Therefore, the application provides a data object brushing method, which includes: when detecting that a client modifies a data segment in a data object in a solid state disk, determining that the data object is a dirty data object, determining that the data segment is a dirty data segment, and adding the dirty data object to a preset dirty data object queue; determining the dirty data objects from the dirty data object queue, and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions; judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is larger than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data larger than the preset dirty data amount lower brushing threshold value to the mechanical hard disk; marking a dirty data segment corresponding to the dirty data which is flushed down to the mechanical hard disk as a target data segment; adding the flushed dirty data objects to a preset target object queue, and judging whether dirty data exist in the flushed dirty data objects or not, if so, re-executing the step of adding the dirty data objects to a preset dirty data object queue, if not, releasing the data of the target data segment in the data object after being brushed down in the target object queue, therefore, the dirty data objects are divided according to the preset unit length of the mechanical hard disk to obtain a plurality of divided areas, and preferentially swipes dirty data in the partition interval having a total amount of dirty data greater than a dirty data amount swipe threshold, so that, the aggregation degree of the lower brush is increased and the seeking frequency of the mechanical hard disk is reduced by avoiding the scattered dirty data of the lower brush, so that the pressure on the mechanical hard disk is reduced and the service performance is improved.
Fig. 3 is a schematic structural diagram of a data object brushing method disclosed in the present application, and as shown in fig. 3, the present application can specifically include the following:
(1) a service IO read-write module: and the module is used for processing read-write IO of the client, recording the modified data segment in the data object as a dirty data segment when the data object is newly written or modified, and adding the data object into the dirty data object LRU queue.
(2) The LRU management module: for maintaining dirty data object LRU queues and target data object LRU queues, responsible for enqueuing and dequeuing data objects. The data object needs to be added to the dirty data object LRU queue if there is data that has not been flushed, and to the target data object LRU queue if there is data that has been flushed but not released. The same data object may also exist in both LRU queues, i.e. the data object is only partially flushed, i.e. the data object has dirty data segments and also has clean data segments.
(3) The lower brush module: the system is used for taking the dirty data object from the dirty data object LRU queue to perform brushing, and transmitting the object to the LRU queue management module to re-enqueue after brushing is completed.
(4) A dirty data amount statistic module: the system is used for updating the total dirty data amount and the dirty data objects when the data objects are added into or taken out of the dirty data object LRU, and updating the threshold value of the dirty data amount of the down-brushing data segments to the down-brushing module in real time.
Referring to fig. 4, the brushing-down flow specifically includes the following contents:
when the data object is written or modified, the newly written and modified data segment is updated to be a dirty data segment; and adding the data object into a dirty data object LRU queue; the lower brushing module takes the dirty data object from the dirty data object LRU queue; and traversing the dirty data objects by taking the minimum allocation unit of the HDD as a partition interval, if the dirty data amount of a certain interval is larger than the dirty data amount threshold of the data segment which can be brushed down, adding the dirty data in the interval into the context of the brushing-down request, and if not, continuously judging the next segment. After traversing, if the dirty data amount of all the intervals is not more than the dirty data amount threshold of the data segment which can be brushed down, the data segment is indicated to be temporarily incapable of being brushed down, and the data object is added into the dirty data object LRU queue again, otherwise, the brushing-down request context is submitted; flushing dirty data in the context of the flushing request, and marking a data segment corresponding to the flushed dirty data as a target data segment; adding the flushed dirty data object into a target data object LRU queue; judging whether the data object after the brushing is finished has dirty data, if so, adding the dirty data object into the LRU queue again; if not, re-enqueue is not needed, the target data segment part in the target data object LRU queue is deleted from the cache device, and the cache space is released. Therefore, the method and the device increase the aggregation degree of the lower brush and reduce the seeking frequency of the mechanical hard disk by avoiding the lower brush of scattered dirty data, thereby realizing the reduction of the pressure on the mechanical hard disk and the improvement of the service performance.
Correspondingly, the embodiment of the present application further discloses a data object brushing device, as shown in fig. 5, the device includes:
the dirty data object adding module 11 is configured to, when a modification operation of a client on a data object in a solid state disk is detected, determine that the data object is a dirty data object, and add the dirty data object to a preset dirty data object queue;
a dirty data object dividing module 12, configured to determine the dirty data object from the dirty data object queue, and divide the dirty data object according to a preset unit length of a mechanical hard disk to obtain a plurality of divided regions;
a dirty data flushing module 13, configured to determine whether a total amount of dirty data of each data segment in the dirty data object corresponding to each partition interval is greater than a preset dirty data amount flushing threshold, and flush, to the mechanical hard disk, dirty data in the partition interval whose total amount of dirty data is greater than the preset dirty data amount flushing threshold;
a data releasing module 14, configured to add the flushed dirty data object to a preset target object queue, and determine whether dirty data exists in the flushed dirty data object, if so, re-execute the step of adding the dirty data object to the preset dirty data object queue, and if not, release data of the flushed data object in the target object queue.
For more specific working processes of the modules, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
The method comprises the steps that when modification operation of a client on a data object in a solid state disk is detected, the data object is determined to be a dirty data object, and the dirty data object is added to a preset dirty data object queue; determining the dirty data objects from the dirty data object queue, and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions; judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is larger than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data larger than the preset dirty data amount lower brushing threshold value to the mechanical hard disk; adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue. Therefore, the dirty data objects are divided according to the preset unit length of the mechanical hard disk to obtain a plurality of division areas, the dirty data in the division areas with the total dirty data amount larger than the dirty data amount lower brushing threshold value are preferably brushed, and therefore the lower brushing aggregation degree is increased and the mechanical hard disk seek times are reduced by avoiding brushing the lower scattered dirty data, and further the pressure on the mechanical hard disk is reduced and the service performance is improved.
Further, the embodiment of the application also provides electronic equipment. FIG. 6 is a block diagram illustrating an electronic device 20 according to an exemplary embodiment, and the contents of the diagram should not be construed as limiting the scope of use of the present application in any way.
Fig. 6 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present disclosure. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display 23, an input output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. Wherein the memory 22 is adapted to store a computer program, which is loaded and executed by the processor 21, to implement the steps of:
when detecting that a client modifies a data object in a solid state disk, determining that the data object is a dirty data object, and adding the dirty data object to a preset dirty data object queue;
determining the dirty data objects from the dirty data object queue, and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions;
judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is larger than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data larger than the preset dirty data amount lower brushing threshold value to the mechanical hard disk;
adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue.
In some embodiments, the processor, by executing the computer program stored in the memory, may specifically implement the following steps:
when detecting that a client modifies a data segment in a data object in a solid state disk, determining that the data object is a dirty data object, determining that the data segment is a dirty data segment, and adding the dirty data object to a preset dirty data object queue.
In some embodiments, the processor, by executing the computer program stored in the memory, may further include the steps of:
determining the total amount of dirty data and the number of dirty data objects in the dirty data object queue, and determining the quotient of the total amount of dirty data in the dirty data object queue and the number of dirty data objects in the dirty data object queue as a first parameter;
determining the size of a single dirty data object in the dirty data object queue, and determining the quotient of the size of the single dirty data object in the dirty data object queue and the preset unit length as a second parameter;
determining an average dirty data amount corresponding to the divided regions based on a quotient of the first parameter and the second parameter, and determining the average dirty data amount as the preset dirty data amount lower brushing threshold.
In some embodiments, the processor, by executing the computer program stored in the memory, may further include the steps of:
and calculating the average dirty data volume according to a preset time interval, and updating the preset dirty data volume lower brushing threshold value by using the average dirty data volume.
In some embodiments, the processor, by executing the computer program stored in the memory, may further include:
and marking a dirty data segment corresponding to the dirty data which is flushed to the mechanical hard disk as a target data segment.
In some embodiments, the processor, by executing the computer program stored in the memory, may specifically implement the following steps:
and releasing the data of the target data segment in the data object after being brushed down in the target object queue.
In some embodiments, the processor, by executing the computer program stored in the memory, may further include the steps of:
and if the total amount of dirty data of each data segment in the dirty data object corresponding to the partition interval is not greater than the preset dirty data amount lower brushing threshold value, re-executing the step of adding the dirty data object to a preset dirty data object queue.
In this embodiment, the power supply 26 is used for providing an operating voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and an external device, and a communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein; the input/output interface 24 is configured to obtain external input data or output data to the outside, and a specific interface type thereof may be selected according to specific application requirements, which is not specifically limited herein.
In addition, the memory 22 is used as a carrier for resource storage, and may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resource stored thereon may include the computer program 221, and the storage manner may be a transient storage or a permanent storage. The computer program 221 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the data object brushing method performed by the electronic device 20 disclosed in any of the foregoing embodiments.
Further, the embodiment of the application also discloses a computer readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements the data object brushing method disclosed above.
For the specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, which are not described herein again.
The embodiments in the present application are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other, that is, for the apparatus disclosed in the embodiments, since the apparatus corresponds to the method disclosed in the embodiments, the description is simple, and for the relevant parts, the method is referred to the method part.
Those of skill would further appreciate that the various illustrative components and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the components and steps of the various examples have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing describes in detail a data object flushing method, apparatus, device, and storage medium provided by the present application, and specific examples are applied herein to explain the principles and implementations of the present application, and the descriptions of the foregoing examples are only used to help understand the method and core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A data object brushing method, comprising:
when detecting that a client modifies a data object in a solid state disk, determining that the data object is a dirty data object, and adding the dirty data object to a preset dirty data object queue;
determining the dirty data objects from the dirty data object queue, and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions;
judging whether the total amount of dirty data of each data segment in the dirty data object corresponding to each partition space is larger than a preset dirty data amount lower brushing threshold value or not, and brushing the dirty data in the partition space with the total amount of dirty data larger than the preset dirty data amount lower brushing threshold value to the mechanical hard disk;
adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue.
2. The data object down-brushing method according to claim 1, wherein when a modification operation of a client on a data object in a solid state disk is detected, determining that the data object is a dirty data object, and adding the dirty data object to a preset dirty data object queue, includes:
when detecting that a client modifies a data segment in a data object in a solid state disk, determining that the data object is a dirty data object, determining that the data segment is a dirty data segment, and adding the dirty data object to a preset dirty data object queue.
3. The method according to claim 1, wherein before determining whether a total amount of dirty data of each data segment in the dirty data object corresponding to each of the partitioned areas is greater than a preset dirty data amount lower brushing threshold, the method further comprises:
determining the total amount of dirty data and the number of dirty data objects in the dirty data object queue, and determining the quotient of the total amount of dirty data in the dirty data object queue and the number of dirty data objects in the dirty data object queue as a first parameter;
determining the size of a single dirty data object in the dirty data object queue, and determining the quotient of the size of the single dirty data object in the dirty data object queue and the preset unit length as a second parameter;
determining an average dirty data amount corresponding to the divided regions based on a quotient of the first parameter and the second parameter, and determining the average dirty data amount as the preset dirty data amount lower brushing threshold.
4. The data object brushing method of claim 3, further comprising:
and calculating the average dirty data volume according to a preset time interval, and updating the preset dirty data volume lower brushing threshold value by using the average dirty data volume.
5. The data object down-brushing method according to claim 2, wherein after the down-brushing the dirty data in the division areas of which the total amount of dirty data is greater than the preset dirty data amount down-brushing threshold value to the mechanical hard disk, further comprising:
and marking a dirty data segment corresponding to the dirty data which is flushed to the mechanical hard disk as a target data segment.
6. The data object flushing method of claim 5, wherein the releasing the data object flushed from the target object queue comprises:
and releasing the data of the target data segment in the data object after being brushed down in the target object queue.
7. The method according to any one of claims 1 to 6, wherein after determining whether a total amount of dirty data of each data segment in the dirty data object corresponding to each of the partitioned areas is greater than a preset dirty data amount lower brushing threshold, the method further comprises:
and if the total amount of dirty data of each data segment in the dirty data objects corresponding to the partition areas is not larger than the preset dirty data amount lower brushing threshold value, the step of adding the dirty data objects to a preset dirty data object queue is executed again.
8. A data object brushing apparatus, comprising:
the dirty data object adding module is used for determining that the data object is a dirty data object when the modification operation of the client on the data object in the solid state disk is detected, and adding the dirty data object to a preset dirty data object queue;
the dirty data object dividing module is used for determining the dirty data objects from the dirty data object queue and dividing the dirty data objects according to the preset unit length of the mechanical hard disk to obtain a plurality of divided regions;
the dirty data brushing module is used for judging whether the total dirty data amount of each data segment in the dirty data object corresponding to each partition interval is larger than a preset dirty data amount brushing threshold value or not, and brushing the dirty data in the partition interval with the total dirty data amount larger than the preset dirty data amount brushing threshold value to the mechanical hard disk;
and the data release module is used for adding the flushed dirty data object to a preset target object queue, judging whether dirty data exists in the flushed dirty data object, if so, re-executing the step of adding the dirty data object to the preset dirty data object queue, and if not, releasing the data of the flushed data object in the target object queue.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the data object brushing method of any of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements a data object brushing method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210868345.3A CN115113824A (en) | 2022-07-22 | 2022-07-22 | Data object brushing method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210868345.3A CN115113824A (en) | 2022-07-22 | 2022-07-22 | Data object brushing method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115113824A true CN115113824A (en) | 2022-09-27 |
Family
ID=83334861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210868345.3A Pending CN115113824A (en) | 2022-07-22 | 2022-07-22 | Data object brushing method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115113824A (en) |
-
2022
- 2022-07-22 CN CN202210868345.3A patent/CN115113824A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3229142B1 (en) | Read cache management method and device based on solid state drive | |
| US8521986B2 (en) | Allocating storage memory based on future file size or use estimates | |
| US8762667B2 (en) | Optimization of data migration between storage mediums | |
| CN108984104B (en) | Method and apparatus for cache management | |
| CN111857589A (en) | SSD cache down-flushing speed control method and system in distributed storage system | |
| EP4020153A1 (en) | Cache space management method and device | |
| CN111309644B (en) | Memory allocation method and device and computer readable storage medium | |
| CN112269661B (en) | Partition migration method and device based on Kafka cluster | |
| CN109977074B (en) | HDFS-based LOB data processing method and device | |
| CN110888846B (en) | Data memory management method and system | |
| CN115113824A (en) | Data object brushing method, device, equipment and medium | |
| CN110941436A (en) | Data storage method and device of application program, storage medium and terminal equipment | |
| CN111221468A (en) | Storage block data deleting method and device, electronic equipment and cloud storage system | |
| CN110795034B (en) | Data migration method, device and equipment of storage system and readable storage medium | |
| CN115840635A (en) | Computing resource management method, electronic device, and program product | |
| CN110362769B (en) | Data processing method and device | |
| CN112446490A (en) | Network training data set caching method, device, equipment and storage medium | |
| JP2009282664A (en) | Storage data management system and storage data management program | |
| CN106557435B (en) | Cache mode identification method, cache module and storage system | |
| CN115145470A (en) | File storage method and system | |
| CN105740167A (en) | File system cache deletion method and system | |
| CN109992211A (en) | A kind of file data management method and relevant apparatus | |
| CN116955212A (en) | Memory defragmentation method, device, equipment and storage medium | |
| CN112905344A (en) | Resource scheduling method, device, equipment and medium in storage system | |
| CN114780014A (en) | Method, electronic device and computer program product for managing metadata storage unit |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |