CN117891416B - Unmapping operation optimization method and device based on SCSI protocol and readable medium - Google Patents

Unmapping operation optimization method and device based on SCSI protocol and readable medium Download PDF

Info

Publication number
CN117891416B
CN117891416B CN202410305298.0A CN202410305298A CN117891416B CN 117891416 B CN117891416 B CN 117891416B CN 202410305298 A CN202410305298 A CN 202410305298A CN 117891416 B CN117891416 B CN 117891416B
Authority
CN
China
Prior art keywords
command
optimized
size
unmapped
optimization
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202410305298.0A
Other languages
Chinese (zh)
Other versions
CN117891416A (en
Inventor
王成强
舒继武
朱博弘
高聪明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xiamen University
Original Assignee
Xiamen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xiamen University filed Critical Xiamen University
Priority to CN202410305298.0A priority Critical patent/CN117891416B/en
Publication of CN117891416A publication Critical patent/CN117891416A/en
Application granted granted Critical
Publication of CN117891416B publication Critical patent/CN117891416B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061Improving I/O performance
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659Command handling arrangements, e.g. command buffers, queues, command scheduling
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0662Virtualisation aspects

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a method, a device and a readable medium for optimizing unmapping operation based on a SCSI protocol, which relate to the field of storage and comprise the following steps: monitoring a SCSI command stream sent to a disk, filtering and checking the SCSI command stream to determine a to-be-optimized unmapped command, and calculating to obtain delay caused by optimizing the to-be-optimized unmapped command; calculating and maintaining environmental factors with great influence on the optimization of the unmapped command to be optimized, wherein the environmental factors comprise disk load, threshold value, IO size and bandwidth; determining a selected optimization strategy, determining the corresponding division size and interval time of a to-be-optimized cancellation mapping command by combining environmental factors, dividing the to-be-optimized cancellation mapping command according to the corresponding division size to obtain a plurality of subcommands, and transmitting each subcommand one by one according to the corresponding interval time until all the subcommands are completely transmitted, thereby solving the problems of delay and performance caused by excessive cancellation mapping block numbers in various complex environments.

Description

Unmapping operation optimization method and device based on SCSI protocol and readable medium
Technical Field
The present invention relates to the field of storage, and in particular, to a method and apparatus for optimizing unmapping operation based on SCSI protocol, and a readable medium.
Background
Storage virtualization technology provides convenient, fast, and accurate storage management functions. By allocating unused space of the previous dedicated disk to other users, i.e., by sharing the disk model, it solves to some extent the problem of underutilization of storage space in conventional approaches. However, this still does not fully exploit the potential of storage virtualization technology. The space allocated to the user is often underutilized. For example, a user may request 500 GB a space, but use less than 20a GB a space. This has led to the development of thin provisioning techniques that allocate only the space that the user actually occupies and allocate additional space from the storage pool as needed. However, this approach still requires a unmap command for space reclamation; otherwise, the disk space will gradually increase, and finally become a standard virtual disk.
The problem is that in conventional mode the unmap operation is treated as a conventional SCSI command without any specific optimization. This approach can lead to several problems:
1. delay and performance issues: in experiments, it was observed that using a demapping operation in the conventional mode resulted in significant performance and delay problems. The severity of these problems is proportional to the number of unmapped blocks. When a demapping command is sent to the disk, the physical disk shows blocking behaviors of different degrees according to the number of demapping blocks; the more blocks to be unmapped, the longer the duration of the blocking.
Experiments in the conventional mode used 1.2 TB HGST SAS SSD and the Fio tool was used to perform sequential read tests on the space of the first 500: 500 GB for 100 seconds. At 10 seconds, a unmap command is sent using Sg3utils to reclaim 500 GB disk space. As shown in fig. 1, rectangular blocks representing standard SCSI commands can only be queued up later when SSD handles triangle blocks representing unmapped commands. The larger triangle blocks represent longer demapping execution times, and longer latency and performance problems. As shown in FIG. 2, it is apparent that when a unmap command is issued to disk and executed, the Fio test is directly blocked and the IOPS performance drops to zero. The performance of the Fio test can only be restored after the unmap command is completed. As shown in fig. 2 (a), the use of the demap command causes extremely high tail delay compared to the original state. As can be seen from fig. 2 (b), the IOPS performance drops to zero during the unmap command execution.
Thus creating a dilemma: if one wants to reclaim a certain space, it is inevitable that other users who share the disk at the time will be affected. Thus, finding a way to quickly reclaim space while minimizing the impact on other users sharing the disk is a problem that must be addressed.
2. There is no unified solution: in practical use of storage virtualization technology, real world scenarios tend to be complex. Virtual disks may consist of a single physical disk or multiple physical disks, as is common in various RAID configurations. In addition, the load state of the underlying physical disk and the size of the I/O operations performed tend to change over time. Furthermore, there is no unified standard to support the demapping property; the support for this function varies greatly from manufacturer to manufacturer, even between different models of the same manufacturer. For example, the Sg3 utils tool may be used to obtain unmap function support data for certain products from INTEL and HGST. Different hard disk drives have significant differences in the unmapping function support, and these different scenarios may affect the optimization effort, so ensuring that the optimization strategy operates stably and efficiently under all these conditions is a challenge.
Disclosure of Invention
The application aims to provide a method, a device and a readable medium for optimizing unmapping operation based on a SCSI protocol, aiming at the technical problems, and the problems of delay and performance caused by excessive number of unmapping blocks in various complex environments are effectively solved.
In a first aspect, the present invention provides a method for optimizing unmapping operations based on SCSI protocol, comprising the steps of:
Monitoring a SCSI command stream sent to a disk, filtering and checking the SCSI command stream to determine a to-be-optimized unmapped command, and calculating to obtain delay caused by optimizing the to-be-optimized unmapped command;
Calculating and maintaining environmental factors with great influence on the optimization of the to-be-optimized unmapped command, wherein the environmental factors comprise disk load, threshold value, IO size and bandwidth, the threshold value is dynamically adjusted according to delay caused by the optimization of the to-be-optimized unmapped command in the previous round, the disk load and the IO size are obtained by analyzing and calculating based on a SCSI command stream, and the bandwidth is obtained by performance test;
Determining a selected optimization strategy, determining a partition size and interval time corresponding to a to-be-optimized cancellation mapping command according to the optimization strategy and environmental factors, partitioning the to-be-optimized cancellation mapping command according to the corresponding partition size to obtain a plurality of subcommands, and transmitting each subcommand one by one according to the corresponding interval time until all the subcommands are completely transmitted.
Preferably, filtering and checking the SCSI command stream to determine the unmapped command to be optimized specifically includes:
Filtering and identifying all unmapped commands in the SCSI command stream;
Each unmap command is checked for unmap commands with a block count greater than a threshold or less than a maximum number of unmap block descriptors as unmap commands to be optimized.
Preferably, the delay caused by optimizing the to-be-optimized unmapped command is calculated, and specifically includes:
determining the time when the to-be-optimized unmapped command enters a sending queue and the time when the to-be-optimized unmapped command enters a response queue;
And subtracting the time of the to-be-optimized canceling mapping command entering the sending queue from the time of the to-be-optimized canceling mapping command entering the response queue to obtain delay caused by optimizing the to-be-optimized canceling mapping command.
Preferably, the calculating and maintaining the environmental factors having a great influence on the optimization of the unmapped command to be optimized specifically include:
counting the size (I) of each I/O command and the I/O total number (iocount (I)) in the SCSI command stream sent to disk I within the time window;
the disk load and IO size were calculated separately using the following formulas:
Wherein, Is the sum of the sizes of all I/O commands,/>For the disk load to be present,For IO size,/>Bandwidth, t is time window;
Setting an initial value of a threshold value, and doubling the threshold value of the current round in response to determining that the delay caused by the optimization of the unmapped command to be optimized in the previous round is less than an acceptable minimum delay value;
in response to the optimization of the unmapped command to be optimized for the previous round causing a delay greater than an acceptable maximum delay, determining a threshold value for the current round using an intermediate value between one half of the threshold value and the threshold value;
Stopping calculation of the delay caused by the optimization of the unmapped command to be optimized for the current round in response to the delay caused by the optimization of the unmapped command to be optimized for the previous round being between an acceptable minimum delay and an acceptable maximum delay;
and calculating and updating the disk load and IO size at preset intervals.
Preferably, the optimization strategy includes a flexible strategy and a default strategy.
Preferably, determining the partition size and the interval time corresponding to the to-be-optimized unmapped command according to the optimization strategy and the environmental factors specifically includes:
in response to determining that the optimization strategy is a flexible strategy, its corresponding partition size and interval time are calculated using the following formulas:
Wherein, Representing partition size,/>Represents the time interval, in ms,Representing a threshold value/>Representing the original size of the unmap command to be optimized,/>Representing the optimal unmapping granularity obtained from the physical device,
In response to determining that the optimization strategy is the default strategy, its corresponding partition size is a threshold and the time interval is 1ms.
In a second aspect, the present invention provides a device for optimizing unmapping operations based on SCSI protocol, comprising:
the monitoring module is configured to monitor a SCSI command stream sent to the disk, filter and check the SCSI command stream to determine a unmapped command to be optimized, and calculate delay caused by optimizing the unmapped command to be optimized;
the parameter calculation maintenance module is configured to calculate and maintain environmental factors with great influence on the optimization of the to-be-optimized unmapped command, wherein the environmental factors comprise disk load, threshold value, IO size and bandwidth, the threshold value is dynamically adjusted according to delay caused by the optimization of the to-be-optimized unmapped command in the previous round, the disk load and the IO size are obtained by analysis and calculation based on a SCSI command stream, and the bandwidth is obtained by performance test;
the optimizing module is configured to determine the selected optimizing strategy, determine the partition size and the interval time corresponding to the canceling mapping command to be optimized according to the optimizing strategy and the environmental factors, partition the canceling mapping command to be optimized according to the corresponding partition size to obtain a plurality of subcommands, and send each subcommand one by one according to the corresponding interval time until all the subcommands are sent completely.
In a third aspect, the present invention provides an electronic device comprising one or more processors; and storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method as described in any of the implementations of the first aspect.
In a fifth aspect, the invention provides a computer program product comprising a computer program which, when executed by a processor, implements a method as described in any of the implementations of the first aspect.
Compared with the prior art, the invention has the following beneficial effects:
(1) The unmapping operation optimizing method based on the SCSI protocol provided by the invention can prevent the blocking from happening by splitting the unmapping command exceeding a certain threshold value in view of the fact that the duration of the blocking is in direct proportion to the number of unmapping blocks. The granularity and spacing of these splits can be adjusted on a case-by-case basis to minimize the impact on performance.
(2) The invention provides an optimization framework for a unmapping operation based on a SCSI protocol, which can be used for carrying out real-time dynamic analysis according to a specific scene and carrying out analysis and calculation by utilizing various environmental factors so as to assist in the optimization process.
(3) The unmapping operation optimizing method based on the SCSI protocol provided by the invention finds the unmapping command to be optimized through the monitoring module, and optimizes the unmapping command to be optimized through the calculation and maintenance of the dynamically changing environmental factors by the auxiliary optimizing module, so that not only can a good optimizing effect be obtained under the comprehensive load and the actual load, but also the problem that other normal IO can be blocked during unmapping is solved, and meanwhile, the influence on the performance is reduced to the greatest extent.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a conventional mode in the prior art;
FIG. 2 is a schematic diagram of the effect of a demapping command on tail latency and performance in a conventional mode in the prior art, wherein 2 (a) is a schematic diagram of the effect of a demapping command on tail latency in a conventional mode, and 2 (b) is a schematic diagram of the effect of a demapping command on IOPS performance in a conventional mode;
FIG. 3 is a flowchart illustrating a method for optimizing a unmapping operation based on a SCSI protocol according to an embodiment of the present application;
FIG. 4 is a schematic diagram of an optimization framework of a SCSI protocol based unmapping operation optimization method according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a monitor module of a method for optimizing unmapping operations based on a SCSI protocol according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a processing center of a method for optimizing unmapping operations based on a SCSI protocol according to an embodiment of the present application;
FIG. 7 is a flow chart corresponding to an optimizing module of a method for optimizing unmapping operations based on a SCSI protocol according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a device for optimizing unmapping operations based on SCSI protocol according to an embodiment of the present application;
fig. 9 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 3 shows a method for optimizing unmapping operations based on SCSI protocol according to an embodiment of the present application, including the following steps:
s1, monitoring a SCSI command stream sent to a disk, filtering and checking the SCSI command stream to determine a to-be-optimized unmapped command, and calculating to obtain delay caused by optimizing the to-be-optimized unmapped command.
In a specific embodiment, filtering and checking the SCSI command stream to determine the unmapped command to be optimized specifically includes:
Filtering and identifying all unmapped commands in the SCSI command stream;
Each unmap command is checked for unmap commands with a block count greater than a threshold or less than a maximum number of unmap block descriptors as unmap commands to be optimized.
In a specific embodiment, the delay caused by optimizing the to-be-optimized unmapped command is calculated, and specifically includes:
determining the time when the to-be-optimized unmapped command enters a sending queue and the time when the to-be-optimized unmapped command enters a response queue;
And subtracting the time of the to-be-optimized canceling mapping command entering the sending queue from the time of the to-be-optimized canceling mapping command entering the response queue to obtain delay caused by optimizing the to-be-optimized canceling mapping command.
In particular, referring to fig. 4 and 5, an embodiment of the present application proposes an optimization framework including a processing center, a monitoring module, and an optimization module, where the monitoring module includes a filtering component, a checking component, and a statistics component. In the filter component, the rectangular blocks represent conventional SCSI commands and the triangular blocks represent unmap commands. The filtering component identifies unmapped commands in all SCSI command streams and sends them to the checking component. The specific detection process comprises the following steps: the filtering component filters out all unmapped commands and then sends them all to the checking component, which finds unmapped commands where the block count is greater than a threshold (Thresholdpoint), less than the maximum number of unmapped block descriptors (this value is obtained from the physical disk), and sends them to the optimization module. The no-process issue with a block count less than the threshold, the command with a block count greater than the maximum number of unmapped block descriptors is discarded and an error is returned. The checking component internally maintains a list of unmapped commands, each of which must be checked by the checking component before being sent to the optimization module. The monitoring module also includes a statistics component responsible for delay statistics. The statistics component maintains a list of delays, wherein each element in each list of delays contains a delay and a corresponding disk ID. The specific tasks are as follows:
(1) Task 1: the SCSI command stream received in the system is continually monitored and unmapped commands that have a significant impact on performance, i.e., unmapped commands to be optimized, are determined. After the filter component recognizes all the unmap commands, these unmap commands are provided to the inspection component for basic inspection. It is an aim of embodiments of the present application to relax user constraints, allowing a user to concentrate on his own needs without taking into account excessive disk constraints. Presently, embodiments of the present application no longer limit the maximum number of unmapped blocks, and unmapped commands with block counts greater than the maximum number of unmapped block descriptors will not return an error. The directly issued unmapped commands with block counts less than the threshold are not intercepted and those unmapped commands with block counts greater than the threshold are sent to the optimization module for optimization.
(2) Task 2: the statistics component portion calculates the delay caused by the optimization of the unmap command to be optimized. This time is counted from the time when the unmap command to be optimized enters the send queue until the corresponding completion command enters the response queue, that is, the time when a cancel operation command to be optimized is entered into the send queue is taken as the start time and the time when the cancel operation command to be optimized appears in the response queue is taken as the end time. Delay caused by the optimization of the unmap command to be optimized = end time-start time. The delay caused by the optimization of the unmap command to be optimized helps calculate the threshold for each disk. After calculating the threshold for disk i, a stop (i) command is sent to the monitor module, which then no longer collects the latency data associated with disk i.
S2, calculating and maintaining environmental factors with great influence on the optimization of the to-be-optimized unmapped command, wherein the environmental factors comprise disk load, threshold value, IO size and bandwidth, the threshold value is dynamically adjusted according to delay caused by the optimization of the to-be-optimized unmapped command in the previous round, the disk load and the IO size are obtained by analyzing and calculating based on a SCSI command stream, and the bandwidth is obtained by performance test.
In a specific embodiment, calculating and maintaining environmental factors that have a significant impact on the optimization of the unmap command to be optimized specifically includes:
counting the size (I) of each I/O command and the I/O total number (iocount (I)) in the SCSI command stream sent to disk I within the time window;
the disk load and IO size were calculated separately using the following formulas:
Wherein, Is the sum of the sizes of all I/O commands,/>For the disk load to be present,For IO size,/>Bandwidth, t is time window;
Setting an initial value of a threshold value, and doubling the threshold value of the current round in response to determining that the delay caused by the optimization of the unmapped command to be optimized in the previous round is less than an acceptable minimum delay value;
in response to the optimization of the unmapped command to be optimized for the previous round causing a delay greater than an acceptable maximum delay, determining a threshold value for the current round using an intermediate value between one half of the threshold value and the threshold value;
Stopping calculation of the delay caused by the optimization of the unmapped command to be optimized for the current round in response to the delay caused by the optimization of the unmapped command to be optimized for the previous round being between an acceptable minimum delay and an acceptable maximum delay;
and calculating and updating the disk load and IO size at preset intervals.
Specifically, referring to fig. 6, in the embodiment of the present application, four parameters, that is, disk load, threshold, IO size, and bandwidth, are maintained in the processing center, and delay caused by optimizing the unmapped command to be optimized, which is transmitted from the monitoring module, is used to calculate the threshold Thresholdpoint [ i ], where i represents the disk number. The contents represented by each of these four parameters are described below.
1. Disk load (load): this is a percentage array that represents the load on the disk over a period of time.
2. IO size (Iosize): this represents the average size of I/O operations on disk during a time window.
3. Threshold (Thresholdpoint): this represents a threshold of disk for screening unmapped commands to be optimized and, in extreme cases, ensuring the rationality of the results of the optimization module. The threshold of the disk indicates that the delay caused by the unmap operation of this size is within an acceptable delay range t min,tmax, which is not noticeable to the user, where t min represents an acceptable minimum delay and t max represents an acceptable maximum delay. Embodiments of the present application suggest and use a range of 0.8ms-1.2 ms.
4. Bandwidth (BW): representing the maximum bandwidth performance of the disk for computing the load. When a disk is attached to the system, the BW array can be determined by a brief, simple performance test, similar to the Fio test. This test measures the maximum bandwidth performance of the newly connected disk to approximate the load state of the disk within a given time window.
Disk load and IO size are both dynamic, varying with time. Thus embodiments of the present application calculate and update the values of disk load and IO size every 10 seconds to maintain their dynamics. To perform these calculations, the necessary information needs to be extracted from the SCSI command stream for analysis. This information includes the size (I) of each I/O command in the SCSI command stream sent to disk I during the time window, and the total I/O count (I) sent to disk I during that time. The above formula is used to calculate the disk load and IO size, where summing size (I) is the sum of the sizes of all I/O commands in the SCSI command stream sent to disk I in this time window, size (I) represents the size of I/O commands in the SCSI command stream at a time, and iocount (I) is the total number of I/Os sent to disk I in this time window.
The initial value of the threshold (Thresholdpoint) is set to 10000, which updates the delay caused by the optimization of the unmap command to be optimized depending on the previous round. If this value is less than t min, the threshold will double, i.e. multiply by 2 at a time. If the value is within the range t min,tmax, it indicates that the target value has been reached and a stop i signal is sent to the monitor module, which then stops the delay calculation for disk i. If the value is greater than t max, a binary search between Thresholdpoint/2 and Thresholdpoint is used to determine the threshold for the current round. For example, the delay caused by the optimization of the unmap command to be optimized for the previous round is 1000ms, t max =800 ms, and the threshold value for the previous round is 100, and the threshold value for the current round is an intermediate value 75 of 50 to 100.
S3, determining the selected optimization strategy, determining the partition size and the interval time corresponding to the to-be-optimized canceling mapping command according to the optimization strategy and the environmental factors, partitioning the to-be-optimized canceling mapping command according to the corresponding partition size to obtain a plurality of subcommands, and transmitting each subcommand one by one according to the corresponding interval time until all the subcommands are transmitted completely.
In particular embodiments, the optimization policies include flexible policies and default policies.
In a specific embodiment, dividing a to-be-optimized cancellation mapping command according to an optimization strategy and environmental factors by a corresponding division size to obtain a plurality of subcommands, and sending each subcommand one by one according to a corresponding interval time, wherein the method specifically comprises the following steps:
in response to determining that the optimization strategy is a flexible strategy, its corresponding partition size and interval time are calculated using the following formulas:
Wherein, Representing partition size,/>Represents the time interval, in ms,Representing a threshold value/>Representing the original size of the unmap command to be optimized,/>Representing the optimal unmapping granularity obtained from the physical device,
In response to determining that the optimization strategy is the default strategy, its corresponding partition size is a threshold and the time interval is 1ms.
In particular, the optimizer is responsible for the actual execution of optimizations that depend on environmental parameters. The optimization module needs to split the unmap command to be optimized into the appropriate size (split size) and send it at the appropriate intervals (interval time) based on this environmental parameter and the selected optimization strategy. Embodiments of the present application provide two optimization strategies: default and flexibility, as described in more detail below.
1) Flexible policy: the split range should not exceed the original size of the unmap command to be optimized (origsize) nor should it be less than the optimal unmap granularity of the disk report to avoid causing excessive additional I/O. When the disk load is high, the partition size should be small and the interval time should be large so as to reduce the influence on the performance to the greatest extent. The size of the cut is also related to Iosize [ i ]. If Iosize [ i ] is large and the cut size is too small, the completion period of the demapping operation can be significantly increased. In summary, the cut size is inversely proportional to the disk load, and proportional to Iosize; the interval time is proportional to the disk load. Furthermore, when Load [ i ] is very small, iosize [ i ]/Load [ i ] becomes very large, resulting in sub-optimal segmentation granularity. Thus, embodiments of the present application require a value to limit this, which is also an important reason for embodiments of the present application to add thresholds. This array keeps the maximum unmap size within an acceptable delay. With the help of the threshold, the unmap command will split into at least Thresholdpoint [ i ] size without significant impact to the user, even under low load conditions.
2) Default policy: in the default strategy, the cut size and interval time need not be calculated. It is divided directly into Thresholdpoint i sizes with 1ms transmission interval. In this case, the Load and Iosize arrays may not need to be maintained.
In summary, the unmap command goes through the following steps from sending to executing: firstly, a demapping command to be optimized is sent to an optimizing module by a monitoring module, and the optimizing module executes specific optimization after obtaining environmental parameters from a processing center. If no stop information for the corresponding disk is received, the monitor module will calculate the delay caused by the optimization of the unmap command to be optimized and send the data to the processing center to update the threshold.
Referring to fig. 7, a de-map command, e.g., length 1000, is initiated to confirm the partition size and partition time according to the optimization strategy. Then, the length 1000 demapping command is split into sub-commands of a split size length, e.g., 10 sub-commands of a length of 100, and then sent one by one at intervals, e.g., every 1ms, ten times. And the demap command starts from a start position, each subcommand being of a split size in length. The segmentation and transmission procedure after selecting the optimization strategy and determining the segmentation size and the interval time is exemplified as follows:
The initial block of the demapping command to be optimized is 2000, the number of blocks is 1000, namely, the demapping command to be optimized starts to operate from the 2000 th block of the magnetic disk, and the number of blocks is 1000, namely, the operation object of the demapping command initial block to be optimized is the 2000 th to 3000 th block of the magnetic disk.
Initial number of demaps = number of blocks = 1000;
Initial start position = start block = 2000, the segmentation size according to the optimization strategy is 100, the following steps are performed:
the segmentation and transmission procedure of the first demapping command is as follows:
1. the first demapping command starts at 2000 a split size of 100 a;
2. the start position of the second demapping command is 2000+100=2100;
3. the second number of demaps is 1000-100=900;
4. Sleep for an interval.
Then, the method further comprises the steps of judging the number of unmapped messages=900 >0, repeating the steps, and dividing and sending the second subcommand as follows:
1. the second demap command starts at 2100 for a split size of 100;
2. the starting position of the third demapping command is 2100+100=2200;
3. the third number of demaps is 900-100=800;
4. Sleep for an interval.
And so on, when the last number of demaps is less than or equal to 0, the process may be stopped.
The above steps S1-S3 do not necessarily represent the order between steps, but the step symbols indicate that the order between steps is adjustable.
With further reference to fig. 8, as an implementation of the method shown in the foregoing figures, the present application provides an embodiment of a device for optimizing a demapping operation based on a SCSI protocol, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 1, and the device is particularly applicable to various electronic devices.
The embodiment of the application provides a unmapping operation optimizing device based on a SCSI protocol, which comprises the following steps:
the monitoring module 1 is configured to monitor a SCSI command stream sent to the disk, filter and check the SCSI command stream to determine a unmapped command to be optimized, and calculate a delay caused by optimizing the unmapped command to be optimized;
The parameter calculation maintenance module 2 is configured to calculate and maintain environmental factors with great influence on the optimization of the to-be-optimized unmapped command, wherein the environmental factors comprise disk load, threshold value, IO size and bandwidth, the threshold value is dynamically adjusted according to delay caused by the optimization of the to-be-optimized unmapped command in the previous round, the disk load and the IO size are obtained by analysis and calculation based on a SCSI command stream, and the bandwidth is obtained by performance test;
The optimizing module 3 is configured to determine the selected optimizing strategy, determine the partition size and interval time corresponding to the canceling mapping command to be optimized according to the optimizing strategy and the environmental factors, partition the canceling mapping command to be optimized according to the corresponding partition size to obtain a plurality of subcommands, and send each subcommand one by one according to the corresponding interval time until all the subcommands are sent completely.
Fig. 9 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention. As shown in fig. 9, the electronic apparatus of the present embodiment includes: a processor 901 and a memory 902; wherein the memory 902 is configured to store computer-executable instructions; a processor 901 for executing computer-executable instructions stored in a memory to implement the steps performed by the electronic device in the above embodiments. Reference may be made in particular to the relevant description of the embodiments of the method described above.
Alternatively, the memory 902 may be separate or integrated with the processor 901.
When the memory 902 is provided separately, the electronic device further comprises a bus 903 for connecting the memory 902 and the processor 901.
The embodiment of the invention also provides a computer storage medium, wherein computer execution instructions are stored in the computer storage medium, and when a processor executes the computer execution instructions, the method is realized.
The embodiment of the invention also provides a computer program product, comprising a computer program, which realizes the method when being executed by a processor.
In the embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described device embodiments are merely illustrative, e.g., the division of modules is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple modules may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.
In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each module may exist alone physically, or two or more modules may be integrated in one unit. The units formed by the modules can be realized in a form of hardware or a form of hardware and software functional units.
The integrated modules, which are implemented in the form of software functional modules, may be stored in a computer readable storage medium. The software functional modules described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or processor to perform some of the steps of the methods of the various embodiments of the application.
It should be appreciated that the Processor may be a central processing unit (Central Processing Unit, abbreviated as CPU), or may be other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, abbreviated as DSP), application SPECIFIC INTEGRATED Circuit (ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile memory NVM, such as at least one magnetic disk memory, and may also be a U-disk, a removable hard disk, a read-only memory, a magnetic disk or optical disk, etc.
The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, an external device interconnect (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of the present application are not limited to only one bus or to one type of bus.
The storage medium may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an Application SPECIFIC INTEGRATED Circuits (ASIC). It is also possible that the processor and the storage medium reside as discrete components in an electronic device or a master device.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.

Claims (7)

1. The unmapping operation optimizing method based on the SCSI protocol is characterized by comprising the following steps:
Monitoring a SCSI command stream sent to a disk, filtering and checking the SCSI command stream to determine a to-be-optimized unmapped command, and calculating to obtain delay caused by optimizing the to-be-optimized unmapped command;
Calculating and maintaining environment factors which have great influence on the optimization of the to-be-optimized unmapped command, wherein the environment factors comprise disk load, threshold value, IO size and bandwidth, the threshold value is dynamically adjusted according to delay caused by the optimization of the to-be-optimized unmapped command in the previous round, the disk load and IO size are analyzed and calculated based on the SCSI command stream, and the bandwidth is obtained through performance test, and the method specifically comprises the following steps:
counting the size (I) of each I/O command and the I/O total number (iocount (I)) in the SCSI command stream sent to disk I within the time window;
the disk load and IO size are calculated separately using the following formulas:
Wherein, Is the sum of the sizes of all I/O commands,/>For the disk load to be present,For IO size,/>Bandwidth, t is time window;
Setting an initial value of the threshold, and doubling the threshold of the current round in response to determining that the delay caused by the optimization of the to-be-optimized unmapped command of the previous round is less than an acceptable minimum delay value;
Determining a threshold value of the current round with an intermediate value between one half of the threshold value and the threshold value in response to the delay caused by the optimization of the unmapped command to be optimized for the previous round being greater than an acceptable delay maximum;
Stopping calculation of the delay caused by the optimization of the unmapped command to be optimized for the current round in response to the delay caused by the optimization of the unmapped command to be optimized for the previous round being between an acceptable minimum delay and an acceptable maximum delay;
calculating and updating the disk load and IO size at intervals of preset time;
Determining a selected optimization strategy, wherein the optimization strategy comprises a flexible strategy and a default strategy, and determining the partition size and the interval time corresponding to the unmapped command to be optimized according to the optimization strategy and the environmental factors, wherein the method comprises the following specific steps:
In response to determining that the optimization strategy is a flexible strategy, its corresponding partition size and interval time are calculated using the following formulas:
Wherein, Representing partition size,/>Represents the time interval, in ms,Representing a threshold value/>Representing the original size of the unmap command to be optimized,/>Representing the optimal unmapping granularity obtained from the physical device,
And responding to the determination that the optimization strategy is a default strategy, wherein the corresponding division size is a threshold value, the time interval is 1ms, dividing the to-be-optimized cancellation mapping command according to the corresponding division size to obtain a plurality of subcommands, and transmitting each subcommand one by one according to the corresponding interval time until all the subcommands are completely transmitted.
2. The method for optimizing a unmapping operation based on a SCSI protocol according to claim 1, wherein said filtering and checking said SCSI command stream to determine a unmapping command to be optimized comprises:
filtering and identifying all unmapped commands in the SCSI command stream;
And checking each demapping command to find out the demapping command with the block count larger than a threshold value or smaller than the maximum demapping block descriptor number as the demapping command to be optimized.
3. The method for optimizing a unmapping operation based on a SCSI protocol according to claim 1, wherein said calculating obtains a delay caused by the optimization of the unmapping command to be optimized, specifically comprising:
Determining the time when the to-be-optimized unmapped command enters a sending queue and the time when the to-be-optimized unmapped command enters a response queue;
And subtracting the time of the to-be-optimized canceling mapping command entering the sending queue from the time of the to-be-optimized canceling mapping command entering the response queue to obtain delay caused by optimizing the to-be-optimized canceling mapping command.
4. A device for optimizing unmapping operations based on SCSI protocol, comprising:
the monitoring module is configured to monitor a SCSI command stream sent to the disk, filter and check the SCSI command stream to determine a to-be-optimized unmapped command, and calculate delay caused by optimizing the to-be-optimized unmapped command;
The parameter calculation maintenance module is configured to calculate and maintain environmental factors having great influence on the optimization of the to-be-optimized unmapped command, wherein the environmental factors comprise disk load, threshold, IO size and bandwidth, the threshold is dynamically adjusted according to delay caused by the optimization of the to-be-optimized unmapped command in the previous round, the disk load and IO size are analyzed and calculated based on the SCSI command stream, and the bandwidth is obtained through performance test, and specifically comprises the following steps:
counting the size (I) of each I/O command and the I/O total number (iocount (I)) in the SCSI command stream sent to disk I within the time window;
the disk load and IO size are calculated separately using the following formulas:
Wherein, Is the sum of the sizes of all I/O commands,/>For the disk load to be present,For IO size,/>Bandwidth, t is time window;
Setting an initial value of the threshold, and doubling the threshold of the current round in response to determining that the delay caused by the optimization of the to-be-optimized unmapped command of the previous round is less than an acceptable minimum delay value;
Determining a threshold value of the current round with an intermediate value between one half of the threshold value and the threshold value in response to the delay caused by the optimization of the unmapped command to be optimized for the previous round being greater than an acceptable delay maximum;
Stopping calculation of the delay caused by the optimization of the unmapped command to be optimized for the current round in response to the delay caused by the optimization of the unmapped command to be optimized for the previous round being between an acceptable minimum delay and an acceptable maximum delay;
calculating and updating the disk load and IO size at intervals of preset time;
The optimizing module is configured to determine a selected optimizing strategy, wherein the optimizing strategy comprises a flexible strategy and a default strategy, and the dividing size and the interval time corresponding to the unmapping command to be optimized are determined according to the optimizing strategy and the environmental factors, and specifically comprises the following steps:
In response to determining that the optimization strategy is a flexible strategy, its corresponding partition size and interval time are calculated using the following formulas:
Wherein, Representing partition size,/>Represents the time interval, in ms,Representing a threshold value/>Representing the original size of the unmap command to be optimized,/>Representing the optimal unmapping granularity obtained from the physical device,
And responding to the determination that the optimization strategy is a default strategy, wherein the corresponding division size is a threshold value, the time interval is 1ms, dividing the to-be-optimized cancellation mapping command according to the corresponding division size to obtain a plurality of subcommands, and transmitting each subcommand one by one according to the corresponding interval time until all the subcommands are completely transmitted.
5. An electronic device, comprising:
one or more processors;
Storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-3.
6. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-3.
7. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-3.
CN202410305298.0A 2024-03-18 2024-03-18 Unmapping operation optimization method and device based on SCSI protocol and readable medium Active CN117891416B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410305298.0A CN117891416B (en) 2024-03-18 2024-03-18 Unmapping operation optimization method and device based on SCSI protocol and readable medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410305298.0A CN117891416B (en) 2024-03-18 2024-03-18 Unmapping operation optimization method and device based on SCSI protocol and readable medium

Publications (2)

Publication Number Publication Date
CN117891416A CN117891416A (en) 2024-04-16
CN117891416B true CN117891416B (en) 2024-05-14

Family

ID=90644529

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410305298.0A Active CN117891416B (en) 2024-03-18 2024-03-18 Unmapping operation optimization method and device based on SCSI protocol and readable medium

Country Status (1)

Country Link
CN (1) CN117891416B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104679668A (en) * 2013-11-26 2015-06-03 群晖科技股份有限公司 Storage system and control method thereof
CN105975358A (en) * 2016-05-03 2016-09-28 深圳市杉岩数据技术有限公司 Fault-tolerant method and system based on SCSI equipment
CN107003942A (en) * 2014-10-27 2017-08-01 桑迪士克科技有限责任公司 To for strengthening the performance of storage device and the processing of persistent unmapped order
WO2019105029A1 (en) * 2017-11-29 2019-06-06 北京忆恒创源科技有限公司 Deallocating command processing method and storage device thereof
CN110825316A (en) * 2018-08-14 2020-02-21 爱思开海力士有限公司 Controller and operation method of the controller
CN111324552A (en) * 2018-12-17 2020-06-23 爱思开海力士有限公司 Memory system and operating method thereof
CN111857598A (en) * 2020-07-28 2020-10-30 山东超越数控电子股份有限公司 Mapping method, device, equipment and readable medium for remote disk localization
CN112328507A (en) * 2019-08-05 2021-02-05 美光科技公司 Memory subsystem for managing flash translation layer table updates based on unmap commands
CN114639418A (en) * 2020-12-15 2022-06-17 美光科技公司 Unmapping operation techniques

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9983992B2 (en) * 2013-04-30 2018-05-29 WMware Inc. Trim support for a solid-state drive in a virtualized environment

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104679668A (en) * 2013-11-26 2015-06-03 群晖科技股份有限公司 Storage system and control method thereof
CN107003942A (en) * 2014-10-27 2017-08-01 桑迪士克科技有限责任公司 To for strengthening the performance of storage device and the processing of persistent unmapped order
CN105975358A (en) * 2016-05-03 2016-09-28 深圳市杉岩数据技术有限公司 Fault-tolerant method and system based on SCSI equipment
WO2019105029A1 (en) * 2017-11-29 2019-06-06 北京忆恒创源科技有限公司 Deallocating command processing method and storage device thereof
CN110825316A (en) * 2018-08-14 2020-02-21 爱思开海力士有限公司 Controller and operation method of the controller
CN111324552A (en) * 2018-12-17 2020-06-23 爱思开海力士有限公司 Memory system and operating method thereof
CN112328507A (en) * 2019-08-05 2021-02-05 美光科技公司 Memory subsystem for managing flash translation layer table updates based on unmap commands
CN111857598A (en) * 2020-07-28 2020-10-30 山东超越数控电子股份有限公司 Mapping method, device, equipment and readable medium for remote disk localization
CN114639418A (en) * 2020-12-15 2022-06-17 美光科技公司 Unmapping operation techniques

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"SAN存储系统的I/O路径控制与管理";付长冬, 舒继武, 吴昊, 沈美明;;《小型微型计算机系统》;20040531;第25卷(第5期);全文 *

Also Published As

Publication number Publication date
CN117891416A (en) 2024-04-16

Similar Documents

Publication Publication Date Title
US9575664B2 (en) Workload-aware I/O scheduler in software-defined hybrid storage system
US8909868B2 (en) Quality of service control method for storage system
US10250684B2 (en) Methods and systems for determining performance capacity of a resource of a networked storage environment
US11792263B2 (en) Methods and systems for managing a resource in a networked storage environment
US20170168729A1 (en) Methods and systems for managing resources of a networked storage environment
US9594515B2 (en) Methods and systems using observation based techniques for determining performance capacity of a resource of a networked storage environment
US11093352B2 (en) Fault management in NVMe systems
US10817348B2 (en) Methods and systems for managing service level objectives in a networked storage environment
US10469582B2 (en) Methods and systems for managing provisioning requests in a networked storage environment
US10210023B2 (en) Methods and systems for managing service level objectives in a networked storage environment
US9465548B1 (en) Methods and systems using model based techniques for determining performance capacity of a resource of a networked storage environment
US20180121237A1 (en) Life cycle management of virtualized storage performance
US9836246B2 (en) Storage management device, performance adjustment method, and computer-readable recording medium
US8458399B2 (en) Methods and structure for determining cache size in a storage system
US20170269854A1 (en) Methods and systems for determining performance capacity of a resource of a networked storage environment
US20240036756A1 (en) Systems, methods, and devices for partition management of storage resources
US10511511B2 (en) Methods and systems for determining performance capacity of a resource of a networked storage environment
US10761726B2 (en) Resource fairness control in distributed storage systems using congestion data
CN117891416B (en) Unmapping operation optimization method and device based on SCSI protocol and readable medium
EP3321809B1 (en) Memory access method, apparatus and system
US10965739B2 (en) Time-based congestion discounting for I/O fairness control
US11055218B2 (en) Apparatus and methods for accelerating tasks during storage caching/tiering in a computing environment
US20220244885A1 (en) Method and system for maximizing performance of a storage system using normalized tokens based on saturation points
US20240143224A1 (en) Systems, methods, and media for prioritizing read accesses to storage devices
CN110874192A (en) Storage management apparatus and storage management method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant