WO2023155703A1

WO2023155703A1 - Workload feature extraction method and apparatus

Info

Publication number: WO2023155703A1
Application number: PCT/CN2023/074657
Authority: WO
Inventors: 金季焜; 鲁鹏; 方维; 黄茜; 张; 李融尚; 史启权; 康宁; 王波超
Original assignee: 华为技术有限公司
Priority date: 2022-02-18
Filing date: 2023-02-06
Publication date: 2023-08-24
Also published as: CN116661675A

Abstract

The present application discloses a workload feature extraction method and apparatus. The method comprises: a storage device obtaining a first workload feature thereof during an input/output (IO) execution process; and storing the first workload feature, the first workload feature being used for memory allocation, data movement, network-attached storage (NAS) load balancing, cold and hot data block identification, prefetching strategy tuning, performance bottleneck perception, load prediction or load change perception of the storage device. By implementing the present application, the storage device extracts the workload feature online, thereby facilitating improvement of the security of user data, and improving the extraction efficiency of the load feature.

Description

A workload feature extraction method and device

technical field

The present application relates to the field of intelligent storage, and in particular to a workload feature extraction method and device.

Background technique

With the development of artificial intelligence technology and the improvement of hardware capabilities, intelligent storage has gradually become a trend. Storage intelligence can better meet users' demands for high capacity, high reliability, high throughput, and low latency, enabling storage devices to achieve self-adaptive optimization in complex business scenarios.

To build an intelligent storage system, the workload information of the storage device is indispensable, and the workload information is an important input for memory allocation, load balancing, data migration and other functional applications. However, currently obtaining workload information is not only inefficient, but also has a great impact on the performance of storage devices, and cannot meet the needs of direct use in the future.

Contents of the invention

The present application discloses a method and device for extracting workload features, which can realize online extraction of workload features of storage devices, effectively improve the security of user data, and also improve the extraction efficiency of load features.

In a first aspect, the present application provides a method for extracting workload features, the method comprising: acquiring a first workload feature of a storage device during an input/output IO execution process; storing the first workload feature, and using the first workload feature Memory allocation of storage devices, data migration, network attached storage NAS load balancing, identification of hot and cold data blocks, prefetch policy tuning, performance bottleneck perception, load forecasting or load change perception.

The foregoing method is applied to a storage device, and the storage device may be, for example, a storage node in a centralized storage system or a storage node in a distributed storage system, which is not specifically limited herein.

It should be noted that the step of obtaining the first workload feature of the storage device during the IO execution process may be performed by a processor (for example, a central processing unit CPU) in the storage device running a software instruction, or it may be independent of Another chip of the processor is executed, and the chip can be an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (graphics processing unit, GPU), embedded neural network processor (neural-network processing units, NPU) and other processing chips. Wherein, the chip may be integrated in the storage device, or placed in the storage device by inserting a card, which is not specifically limited here.

In the above method, the storage device extracts the workload characteristics of the storage device online during the IO execution process, which not only avoids the problem of low security caused by the direct leakage of the user's workload data (for example, IO data), but also improves Extraction efficiency of workload characteristics. In addition, the extracted workload characteristics can be directly used by each module of the storage device itself, so as to realize self-adaptive optimization of the storage device and help improve the intelligence of the storage system.

Optionally, acquiring the first workload feature of the storage device includes: obtaining the first workload feature by a processor of the storage device according to the workload data in the memory.

Implementing the above implementation manner, in the storage device, the processor directly reads the workload data from the memory, and extracts the first workload feature according to the read workload data, realizing the online extraction of the workload feature. Compared with the way that an external device consumes a lot of time to copy all the workload data from the storage device and extract workload features offline based on the workload data, the online workload feature extraction method in this application reduces the need for the storage device itself. performance impact.

Optionally, the feature of the first workload is extracted offline without an external device.

Implementing the above implementation method, the first workload feature does not need to be extracted offline by an external device, but is extracted online by the storage device itself, which not only reduces the impact on the performance of the storage device, but also improves the extraction efficiency of the workload feature.

Optionally, in the block business scenario, the first workload feature includes at least one of a time feature, a flow feature, and a hotspot distribution feature, where the time feature is used to indicate the time interval corresponding to the IO of the block business, and the flow feature is used for Indicates the access mode of the IO flow of the block service, and the hotspot distribution feature is used to indicate the reuse distance distribution of the address block in the block service.

Wherein, the time feature may be, for example, the total time interval, the distribution of IO time intervals, and the like. The total time interval is used to indicate the total duration corresponding to the batch IO of the block service, and the IO time interval distribution is used to indicate the time interval between IOs in the batch IO. Based on the time characteristics, the access time distribution of IO in the block business can be known.

Stream features include at least one of the following features: number of IO streams, IO stream length distribution, IO stream bandwidth distribution, IO stream interval distribution, IO stream space concurrency distribution, proportion of sequential streams and interval streams in IO streams, and disorder The number of IO streams. The stream characteristics reflect the characteristics of IO stream access in the block business scenario.

The reuse distance of an address block refers to the number of non-duplicated address blocks between two adjacent accesses to the same address block. The distribution of hotspots can reflect the regularity of repeated access to address blocks by IOs of the block business.

Optionally, in the block business scenario, the first workload feature further includes at least one of the following features: IO size distribution, read-write ratio parameters, total read-write bandwidth, and the number of unaligned IOs.

Optionally, in the file system service scenario, the first workload feature includes a short-term access feature, which is used to indicate that the batch IO of the file system service is at least Access patterns in one dimension.

Among them, in the dimension of files, the short-term access features include at least one of the following features: number of files accessed by batch IO, file size distribution, file reuse distance distribution, file concurrent operation number distribution, etc.

In the directory dimension, the short-term access characteristics include at least one of the following characteristics: number of directories accessed by batch IO, directory depth and width distribution, directory reuse distance distribution, total read and write bandwidth distribution in the directory, directory operations Quantity distribution, number distribution of concurrent directory operations, sequential access sequence of files in a directory, etc.

In the dimension of time, the short-term access feature includes at least one of the following features: the total time interval corresponding to the batch of IOs and the distribution of time intervals.

In the dimension of IO in the file, the short-term access characteristics include at least one of the following characteristics: the size distribution of IO in the file, the read and write bandwidth of the IO in the file, the reuse distance distribution of the address block in the file, and the IO in the file. Stream characteristics and the order of IO streams within the file, etc.

In the dimension of operation, the short-term access characteristics include at least one of the following characteristics: the proportion of operation commands, the distribution of host operations, and the distribution of operation modes, where the host operations are identified based on the batch IO of the file system business User operations on the host can also be called aggregation operations, user operations, and so on. The host operation can be, for example, cp (indicating copy), rm (indicating deleting a directory or file), rmdir (indicating deleting an empty directory), grep (indicating query) etc. on the Linux system. Operation modes include but are not limited to sequential read, sequential write, random read, random write, create write, append write, overwrite, file lock, protocol lock, file system lock, etc.

Implement the above implementation method, in the file system business scenario, for batch IO in a short period of time, from the dimensions of file, directory, time, IO and operation in the file, realize the workload analysis in the file system business scenario Multi-angle and multi-level descriptions improve the accuracy of workload characteristics extracted in file system business scenarios.

Optionally, in the file system business scenario, the first workload feature also includes a global feature, the global feature is used to indicate the hierarchical structure distribution of the file system, and the global feature includes at least one of the following features: the number of files in the file system, The number of directories in the file system, the distribution of directory depth in the file system, the distribution of the number of files in the directory in the file system, the distribution of file access frequency in the file system, and the distribution of directory access frequency in the file system.

Implementing the above implementation, the hierarchical structure layout of the file system is described from a global perspective, for example, the number of directories, files, and files in the directory, etc., and the overall access to the directories and files of the file system The frequency enriches the expression of the workload of the file system business and improves the accuracy of the extracted workload characteristics in the file system business scenario.

Optionally, the method further includes: compressing the first workload features, obtaining second workload features and compression parameters, the number of features included in the second workload features is less than the number of features included in the first workload features, and the compression parameters It is used to restore the characteristics of the second workload to the characteristics of the first workload; according to the characteristics of the second workload and compression parameters, perform NAS load balancing, memory allocation, data migration, hot and cold data block identification, prefetch policy optimization, performance bottleneck detection, Load forecasting or load change sensing.

Implementing the above implementation, by compressing the first workload feature, not only the storage space required for storing the workload feature can be reduced, but also the amount of data when transmitting the workload feature can be reduced, and the data exchange between the storage device and the simulation device can be improved. transmission efficiency. In addition, the storage device can directly use the first workload features, such as NAS load balancing, memory allocation, data migration, hot and cold data block identification, prefetch policy optimization, load prediction, etc., to achieve self-adaptive optimization of each module. The intelligence degree of the storage device is improved.

Optionally, the method further includes: sending the second workload characteristics and compression parameters to the simulated device, so that the simulated device performs memory allocation, data movement, NAS load balancing, hot and cold data block identification, Prefetching strategy tuning, performance bottleneck perception, load forecasting or load change perception.

Implementing the above implementation method, the simulation device can implement various applications according to the received second workload characteristics and compression parameters, such as memory allocation, data migration, NAS load balancing, etc., and realize offline testing and scene simulation of various scenarios.

Optionally, the second workload feature and compression parameters are also used for the simulation device to acquire IO simulation data and verify whether the first workload feature is credible based on the IO simulation data.

For example, the simulation device can first obtain the first workload feature based on the second workload feature and compression parameters, obtain IO simulation data according to the first workload feature, and further verify whether the first workload feature is credible according to the IO simulation data. Specifically, according to the IO simulation Re-extract the third workload feature from the data, and compare whether the third workload feature is consistent with the first workload feature. When the third workload feature is consistent with the first workload feature, it means that the first workload feature is more credible, and the IO simulation data can be used as a user The real workload data of the device enables the reproduction of user scenarios.

Implementing the above implementation method, the simulation device can further prove the extracted first workload feature. When it is determined that the first workload feature is credible, the simulation device can successfully obtain the user device through simulation using the first workload feature transmitted by the storage device. on-site data.

In a second aspect, the present application provides a workload feature extraction device, which includes a processing unit, used to acquire the first workload feature of the storage device during the execution of input and output IO; a storage unit, used to store The first workload feature, the first workload feature is used for memory allocation of storage devices, data migration, network attached storage NAS load balancing, hot and cold data block identification, prefetch policy optimization, performance bottleneck perception, load prediction or load change perception.

Optionally, the processing unit is specifically configured to: use the processor to obtain the first workload feature according to the workload data in the memory.

Optionally, in the file system business scenario, the first workload feature also includes a global feature, the global feature is used to indicate the hierarchical structure distribution of the file system, and the global feature includes at least one of the following features: the number of files in the file system, the number of files The number of directories in the system, the directory depth distribution of the file system, the number of files in the directory of the file system, the file access frequency distribution of the file system, and the directory access frequency distribution of the file system.

Optionally, the processing unit is further configured to: compress the first workload feature, obtain the second workload feature and compression parameters, the number of features included in the second workload feature is less than the number of features included in the first workload feature, and compress the parameter It is used to restore the characteristics of the second workload to the characteristics of the first workload; according to the characteristics of the second workload and compression parameters, perform NAS load balancing, memory allocation, data migration, hot and cold data block identification, prefetch policy optimization, performance bottleneck detection, Load forecasting or load change sensing.

Optionally, the apparatus further includes: a sending unit, configured to send the second workload characteristics and compression parameters to the simulation device, so that the simulation device performs memory allocation, data movement, and network attached storage NAS load according to the second workload characteristics and compression parameters. Balancing, identification of hot and cold data blocks, prefetch policy tuning, performance bottleneck perception, load forecasting or load change perception.

In a third aspect, the present application provides a device, which includes a processor and a memory, wherein the memory is used to store program instructions; the processor invokes the program instructions in the memory, so that the device executes the first aspect or the first aspect. A method in any possible implementation of an aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed by a processor, the above-mentioned first aspect or any possible implementation of the first aspect can be realized. method.

In a fifth aspect, the present application provides a computer program product. When the computer program product is executed by a processor, the method in the above-mentioned first aspect or any possible embodiment of the first aspect is implemented. The computer program product can be, for example, a software installation package. If the method provided by any possible design of the first aspect above needs to be used, the computer program product can be downloaded and executed on the processor. , so as to implement the first aspect or the method in any possible embodiment of the first aspect.

In a sixth aspect, the present application provides a system, the system includes a processor, a chip, and a memory, wherein the processor and/or the chip is used to obtain the first workload workload feature of the storage device during the execution of the input and output IO ;The memory is used to store the first workload feature, and the first workload feature is used for memory allocation of storage devices, data movement, network attached storage NAS load balancing, hot and cold data block identification, prefetch policy optimization, performance bottleneck perception, load prediction or load change sensing.

Optionally, the processor and/or the chip is further configured to compress the first workload features to obtain second workload features and compression parameters, the number of features included in the second workload features is less than the number of features included in the first workload features, Compression parameters are used to restore the second workload characteristics to the first workload characteristics; according to the second workload characteristics and compression parameters, perform NAS load balancing, memory allocation, data migration, hot and cold data block identification, prefetch policy tuning, Performance bottleneck awareness, load forecasting, or load change awareness.

For specific content about the first workload feature, refer to the related description of the first workload feature in the first aspect.

The technical effects of the above-mentioned second aspect to the sixth aspect are the same as those of the above-mentioned first aspect, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of modules of a storage device provided in an embodiment of the present application;

FIG. 3 is a flow chart of a workload feature extraction method provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of a block access sequence provided by an embodiment of the present application;

FIG. 5 is a flow chart of a workload feature extraction method provided in an embodiment of the present application;

FIG. 6 is a schematic diagram of a functional structure of a storage device provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a storage device provided by an embodiment of the present application.

Detailed ways

Terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The terms "first" and "second" in the description and claims in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order.

It should be noted that, in the embodiment of the present application, a description such as "at least one (or at least one) of a1, a2, ... and an" is used, including any one of a1, a2, ... and an The case of being alone also includes the case of any combination of any number of a1, a2, ... and an, and each case can exist alone. For example, the description of "at least one of a, b, and c" includes a alone, b alone, c alone, a combination of a and b, a combination of a and c, a combination of b and c, or a combination of abc Condition.

For ease of understanding, the following first introduces related terms and the like that may be involved in the embodiments of the present application.

(1) block storage

Block storage means that data is stored in fixed-size data blocks, and each data block is assigned a number for addressing. Block storage often adopts a storage area network (Storage Area Network, SAN) architecture. SAN is a storage architecture that connects storage devices and application servers through a network, and this network is dedicated to access between hosts and storage devices. When there is a demand for data access, the data can be transmitted at high speed between the server and the background storage device through the storage area network. SAN provides block-level storage services, which can effectively improve data transmission efficiency and read/write speed.

(2) File storage

File storage refers to the way of storing data in the form of files. File storage often adopts a network-attached storage (Network-Attached Storage, NAS) architecture to provide file-level data access and sharing services. In addition to providing file sharing services to users, it can also control user access rights (for example, add, delete, modify, etc.). NAS is implemented by installing a file system on a storage device and sharing storage space in the form of a file directory. The feature of NAS is that it includes a file system and an operating system, and can run completely independently. It is a file-level shared storage device with low cost and integrated software and hardware.

Using the concept of "file" to organize data in the computer, data for the same purpose can be composed of different types of files according to the structure required by different applications. Different suffixes are usually used to refer to different types, and each file has a corresponding file name. And when there are many files, the files can be grouped, and each group of files is placed in the same directory (or folder). In addition, there may be a subdirectory (subdirectory or subfolder) under the directory except files, and all files and directories form a tree structure. This tree structure is called: File System (File System), the file system defines the necessary data structure and disk data management methods when storing files on disk. There are many forms of file systems, for example, FAT/FAT32/NTFS of Windows, EXT2/EXT3/EXT4/XFS/BtrFS of Linux, hdfs file system of Hadoop, etc.

(3) Reuse distance

The reuse distance (RD) refers to the number of unique data separated between two adjacent accesses to the same memory data. The reuse distance from the current visit to the next visit is called the forward reuse distance (next reuse distance, NRD), and the reuse distance from the current visit to the last visit is called the backward reuse distance (previous reuse distance, PRD). Unless otherwise specified, the reuse distance generally refers to the forward reuse distance.

(4) Input and output IO stream

A stream is a sequence of bytes with a starting point and an ending point, and information is sent in a first-in, first-out manner. The IO stream can be divided into input stream (InputStream) and output stream (OutputStream) according to the flow direction. The input stream means that the data in the hard disk is input to the memory, and the output stream means that the data in the memory is output to the hard disk. The IO stream can be divided into a byte stream and a character stream according to the size of the data processing unit. The byte stream is a stream for reading and writing data in units of bytes, and the character stream is a stream for reading and writing data in units of characters. In addition, the IO stream may also have other classification methods, which are not specifically limited here.

In an implementation manner, collecting workload data of the user equipment is one of effective ways to perceive the workload of the user equipment. For example, a log storing workload data (for example, IO data) is copied from the storage device, and a workload feature of the storage device is obtained based on the workload data in the log. However, due to the large amount of workload data, the data copy process takes a long time, which not only affects the performance of the storage device itself, but also involves customer privacy and security issues. In addition, the workload characteristics of storage devices obtained based on workload data are relatively simple, such as IO read/write ratio, stream ratio, stream bandwidth, etc., which cannot accurately represent the workload of storage devices.

Aiming at the above-mentioned data collection process affecting device performance, low efficiency, and coarse-grained workload extraction, the embodiment of this application proposes a workload feature extraction method, which can achieve the premise of reducing performance consumption and storage space consumption as much as possible. Under this condition, the workload characteristics of storage devices can be extracted efficiently and accurately, and this method has good applicability.

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application. The system can be used to extract the workload characteristics of the storage device and perform IO flow simulation based on the workload characteristics of the storage device. As shown in FIG. 1 , the system includes a storage device and an emulation device, where the storage device and the emulation device can communicate in a wireless or wired manner.

Wherein, the storage device may be a storage node in a centralized storage system, or a storage node in a distributed storage system. The simulated device may be a device with a computing function, for example, a server deployed on the network side, or a component or chip in the server. The network-side device may be deployed in a cloud environment, that is, a cloud computing server, or the network-side device may also be deployed in an edge environment, that is, an edge computing server. The network side device may be one integrated device, or multiple distributed devices, which is not specifically limited in this embodiment of the present application.

The storage device is used to extract its own workload characteristics online, and based on the extracted workload characteristics, it can perform optimal memory allocation, NAS load balancing, data migration, hot and cold data block identification, prefetch policy optimization, performance bottleneck perception, At least one of operations such as load forecasting or load change sensing. The storage device is also used to send the extracted workload characteristics to the simulation device, so that the simulation device performs IO flow simulation according to the extracted workload characteristics, and reproduces the load scenario of the storage device.

Exemplarily, in order to reduce the amount of data transmitted by the storage device and further improve the security of the storage device data, after the storage device extracts the workload characteristics, it may first compress the extracted workload characteristics to obtain the compressed workload characteristics , the compressed workload features still have the field names of each feature and are interpretable, and then the The compressed workload characteristics are sent to the simulated device.

The simulation device is used to receive the workload characteristics sent by the storage device, and perform IO flow simulation based on the received workload characteristics to generate the IO simulation data of the storage device, so as to realize the reproduction of the real business scenario of the storage device. In some possible embodiments, the simulated device receives the compressed workload characteristics from the storage device, and the simulated device needs to decompress and restore the compressed workload characteristics before performing the IO flow simulation operation to obtain the uncompressed workload characteristics. Compressed workload characteristics. In some possible embodiments, the simulation device can also perform memory allocation, load balancing, data movement, hot and cold data block identification, prefetch policy tuning, performance bottleneck perception, load prediction or load change according to the received workload characteristics perception.

In some possible embodiments, when a storage device is deployed in a distributed storage system, if the storage device is a management node in the distributed storage system, in addition to extracting the workload characteristics of its own device, the storage device is also used to Summarize the workload characteristics of other storage nodes in the distributed storage system, for example, receive the workload characteristics extracted by other storage nodes, and compress the workload characteristics of each storage node separately, and send the compressed workload characteristics to emulated device.

It should be noted that FIG. 1 is only an exemplary architecture diagram, but does not limit the number of network elements included in the system shown in FIG. 1 . Although not shown in FIG. 1 , in addition to the functional entities shown in FIG. 1 , FIG. 1 may also include other functional entities. In addition, the method provided in the embodiment of the present application can be applied to the system shown in FIG. 1 , and of course the method provided in the embodiment of the present application can also be applied to other systems, which is not limited in the embodiment of the present application.

Referring to FIG. 2 , FIG. 2 exemplarily shows a block diagram of a storage device. The storage device can be used to extract workload characteristics. As shown in Figure 2, the storage device at least includes a data acquisition module, an online feature extraction module, a feature query interface and an optimization module.

Wherein, the data collection module is used to obtain the workload data from the memory of the storage device and input the obtained workload data to the online feature extraction module. The online feature extraction module includes a feature calculation module and a feature processing module. The feature calculation module can perform multi-type (for example, block business and file system business) and multi-granularity (for example, logical unit number, file system, etc.) features based on workload data Extract and calculate, obtain the workload feature of the storage device and input the workload feature of the storage device to the feature processing module, and the feature processing module is used to compress the workload feature to obtain the compressed workload feature. It should be noted that, for the specific content of the features of the workload, reference may be made to the relevant descriptions in the following method embodiments, which will not be repeated here.

The feature query interface is used as a unified entrance for the tuning module to access the online feature extraction module, and the feature query interface is used to realize data transmission between the tuning module and the online feature extraction module. For example, the tuning module can obtain workload features through the feature query interface and perform algorithm tuning, such as optimal memory allocation, NAS load balancing, identification of hot and cold data, and optimization of data migration strategies. Wherein, the tuning module includes but is not limited to a memory allocation module, a NAS load regulation module, a garbage collection module, and the like.

Exemplarily, the memory allocation module in the tuning module obtains the workload feature from the online feature extraction module through the feature query interface, and optimizes the resource allocation strategy based on the workload feature to obtain an optimal memory allocation strategy.

In some possible embodiments, after the online feature extraction module obtains the workload features, it may also perform persistent storage on the workload features, or store the workload features in a corresponding database.

In some possible embodiments, the storage device further includes a data return module. The data return module is used to receive the workload feature sent by the online feature extraction module, and send the workload feature to an external device (for example, a simulation device), so that the external device can realize the IO flow simulation of the offline real business based on the workload feature. It is used to assist fault location on the live network, test various offline scenarios, and evaluate algorithm strategies.

It should be noted that the workload feature input by the online feature extraction module to the data return module can be: the workload feature compressed by the feature processing module, or the uncompressed workload feature output by the feature calculation module. The embodiments of the present application do not make specific limitations.

In the embodiment of the present application, each module of the storage device shown in FIG. 2 mainly consumes computing power of a core central processing unit (central processing unit, CPU). In some possible embodiments, as the amount of workload feature data extracted by the storage device increases, the storage device shown in Figure 2 may also include a service distribution deployment module, which is used to implement core CPU computing For example, by distributing and deploying network cards, data processing units (DPUs, DPUs), chips, etc. to storage devices to expand the computing power of storage devices.

It should be noted that the schematic diagram of modules of the storage device shown in FIG. 2 is only an example. In some possible embodiments, the storage device may include more or fewer modules than those shown in FIG. 2 . In addition, each module shown in FIG. 2 may be implemented by software, hardware, or a combination of hardware and software.

Referring to FIG. 3 , FIG. 3 is a flowchart of a workload feature extraction method provided by an embodiment of the present application, which is applied to a storage device. The method includes but is not limited to the following steps:

S101: Acquire a first workload feature of a storage device during an input/output IO execution process.

In the embodiment of the present application, the storage device acquires the first workload feature during the IO execution process, which means that the storage device extracts the first workload feature of its own device in an online manner.

In order to accurately describe the workload of the storage device, the storage device extracts different workload characteristics in different business scenarios. The embodiment of the present application mainly introduces the extraction of workload features of the storage device in two business scenarios, one of which is a block business, and the other is a file system business.

The following describes the workload features extracted in these two business scenarios:

The first type: block business

In this embodiment of the application, in the block business scenario, the first workload features include at least one of time features, flow features, and hotspot distribution features:

(1) Time characteristics:

The time feature is used to indicate the time interval corresponding to the input and output IO of the block service.

In the embodiment of the present application, the time feature specifically includes: a total time interval and a distribution of IO time intervals. Wherein, the total time interval is used to indicate the total duration corresponding to the batch IO of the block service, and the IO time interval distribution is used to indicate the time interval between IOs in the batch IO of the block service.

For example, the total time interval may be 10ms, the batch IO is the number of IOs contained within 10ms, and the IO time interval distribution indicates the time interval of each IO within 10ms. In some possible embodiments, the total time interval may also be 12ms, 20ms or other values.

(2) Flow characteristics

The flow feature is used to indicate the access mode of the IO flow of the block service.

It should be noted that the IO stream includes multiple IOs. In the IO flow, the distance interval between any two adjacent IOs is smaller than a preset interval threshold and the access time interval between any two adjacent IOs is smaller than a preset time threshold. The IO stream can be a sequential stream or an interval stream. Each IO in the sequential stream is a sequential IO. The sequential IO means that the read and write operations access data from adjacent addresses one by one based on the logical block. The sequential stream The distance interval between any two adjacent IOs in the interval stream is 0; each IO in the interval stream can be a random IO. The distance interval between adjacent IOs is not 0. In other words, each IO in the sequential stream is continuous and uninterrupted in logical address, while each IO in the interval stream is intermittent in logical address.

In this embodiment of the application, the stream features include at least one of the following features:

a. Number of IO streams: used to indicate the number of IO streams in the batch IO.

b. IO flow length distribution: used to indicate the length of each IO flow of the batch IO, the length of the IO flow is the IO flow contained The number of IOs.

c. IO flow bandwidth distribution: used to indicate the bandwidth of each IO flow of the batch IO.

d. IO flow interval distribution: used to indicate the distance between each IO flow of a batch of IO.

e. Proportion of sequence flow and interval flow: used to indicate the ratio of the number of sequence flow to the number of interval flow in the batch IO.

f. The number of out-of-order IO streams: used to indicate the number of out-of-order IO streams in the batch IO.

Use a specific example to illustrate the out-of-order IO stream: Assume that the offsets of the starting positions corresponding to each IO in IO stream 1 are 10, 20, 30, 40, ..., that is, for any two adjacent IOs, the next If the offset of the IO is greater than the offset of the previous IO, the IO stream 1 will not be disordered. Assume that the offsets of the starting positions corresponding to each IO in IO stream 2 are 20, 40, 30, 50, 10, ..., that is, the offsets of IOs in IO stream 2 neither show an increasing trend nor present Decreasing trend, then IO flow 2 is an out-of-order IO flow.

g. Spatial concurrency distribution of IO streams: used to indicate the spatial concurrency of IO streams in batch IO.

In a specific implementation, the concurrency distribution of the IO stream space may be expressed as at least one of the following: the distribution of the number of IO streams contained in an address block of a preset size; the number of address blocks of a preset size accessed by an IO stream.

For example, the address block with a preset size can be a block with a size of 4M (abbreviated as 4M block), and the feature of concurrent distribution of IO stream space specifically includes: the distribution of the number of IO streams contained in a 4M block and the IO stream in batch IO The number of 4M blocks accessed.

(3) Distribution characteristics of hot spots

The hot spot distribution feature is used to indicate the reuse distance distribution of the address blocks accessed by the IO in the block business.

In the embodiment of the present application, the hotspot distribution feature includes the distribution of reuse distances of address blocks, and the reuse distance of address blocks refers to the number of non-duplicated address blocks between two adjacent accesses to the same address block. For the calculation of the reuse distance distribution, please refer to the following description about the reuse distance. For the sake of brevity, details are not repeated here.

It should be noted that the reuse distance distribution of address blocks can be classified according to the index. For example, the reuse distance distribution of address blocks with an interval distance of 2048 IO accesses and the reuse distance of address blocks with an interval distance of 4096 IO accesses can be counted sequentially. The distance distribution, the reuse distance distribution of the address blocks accessed by the IO within the interval distance of 8192, ..., so that the distribution of the address blocks frequented by IO can be characterized based on different interval distance scales.

In some possible embodiments, the hotspot distribution feature may also include the number of non-duplicated IOs in the batch of IOs. The number of non-duplicated IOs refers to the number of IOs that access different address blocks.

In some possible embodiments, in the block business scenario, the first workload feature further includes at least one of the following features: IO size distribution, read-write ratio parameter, total read-write bandwidth, number of unaligned IOs, and Second input and output (Input/Output Per Second, IOPS).

Among them, the IO size distribution is used to indicate the size of each IO in the batch IO, the read-write ratio parameter is used to indicate the ratio of the number of read IOs in the batch IO to the number of write IOs, and the number of unaligned IOs is used to indicate The number of IOs with misaligned length and/or the number of IOs with misaligned offset.

It should be noted that unaligned IO may cause read amplification or write amplification during the read and write process, resulting in increased consumption of disk IO, which not only reduces the read and write efficiency of the disk, but also affects the performance of the disk. And IO alignment (that is, length alignment and offset alignment) can effectively save disk IO consumption and improve disk read and write efficiency. It can be seen that the number of unaligned IOs can be used to analyze the performance of the disk.

The second type: file system business

In this embodiment of the application, in the file system business scenario, the first workload feature includes a short-term access feature, which is used to indicate the batch IO of the file system business in the directory, file, time, IO and The access pattern on at least one dimension in the operation.

The following describes the specific content of the short-term access feature from these five dimensions:

(1) Directory

Based on the directory dimension, short-term access characteristics include at least one of the following characteristics: number of directories, distribution of directory depth and width, distribution of directory reuse distance, distribution of total read and write bandwidth in a directory, distribution of metadata operations corresponding to a directory, The distribution of the number of directory operations, the distribution of the number of concurrent directory operations, and the sequential access sequence of files in the directory.

Wherein, the directory depth refers to the length of the subdirectories nested in the directory, and the directory width refers to the number of files in the directory.

The directory reuse distance is used to indicate the number of distinct directories between two consecutive accesses to the same directory. The so-called same directory means that the identifiers (IDs) of the directories are the same, and each directory in the file system has a corresponding directory ID.

Operations on directories include but are not limited to viewing, querying, copying, switching, creating, deleting, cutting, renaming, changing attributes, etc.

The number of concurrent directory operations refers to the number of clients operating on the same directory at the same time.

The sequential access sequence of files in a directory can be used to analyze the correlation between files in the directory, which is beneficial to the prefetching of the sequential flow of files in a directory in the file system business, and improves the hit rate of file access in the directory.

(2) Documents

Based on the dimension of files, the short-term access characteristics include at least one of the following characteristics: file quantity, file size distribution, file reuse distance distribution, file operation quantity distribution and file concurrent operation quantity distribution.

Among them, the number of files includes but is not limited to the total number of files accessed by batch IO, the number of files corresponding to read IO in batch IO, the number of files corresponding to write IO in batch IO, and the number of files corresponding to each operation mode in batch IO , the operation modes include sequential read, random read, sequential write, random write, create write, append write, overwrite write, file lock, etc.

The file reuse distance is used to indicate the number of unique files between two adjacent accesses to the same file. The so-called same file refers to the same file ID, and each file in the file system has a corresponding file ID.

Operations on files include but are not limited to: viewing, querying, copying, switching, creating, deleting, cutting, renaming, changing attributes, etc.

The number of concurrent file operations refers to the number of clients operating on the same file at the same time.

(3) time

Based on the dimension of time, the short-term access feature includes at least one of the following features: total time interval and time interval distribution, wherein the total time interval is used to indicate the total duration corresponding to the batch IO of the file system business, and the time interval The distribution is used to indicate the time interval between each file IO in the batch IO of the file system service.

(4) IO in the file

Based on the dimension of IO in a file, short-term access features include at least one of the following features: IO size distribution in a file, read and write bandwidth of IO in a file, reuse distance distribution of address blocks in a file, and IO flow in a file Characteristic and sequentiality of IO streams within a file.

Wherein, the read-write bandwidth of the IO in the file includes at least one of the total read-write bandwidth and the distribution of the read-write bandwidth of a single file.

The reuse distance of the address block in the file is used to indicate the number of non-duplicated address blocks between two adjacent accesses to the same address block in the same file.

The characteristics of the IO streams in the file include at least one of the number of IO streams in the file, the length distribution of the IO streams in the file, the bandwidth distribution of the IO streams in the file, the interval distribution of the IO streams in the file, and the spatial concurrency distribution of the IO streams in the file. It should be noted that, for details about the characteristics of the IO flow in the file, reference may be made to the relevant description of the above-mentioned flow characteristics of the block service, which will not be repeated here.

The sequence degree of the IO stream in the file is used to indicate the sequence of the IO stream in the file. Exemplarily, the order degree of the IO stream in the file can be obtained by weighted calculation according to the length of the IO stream in the file and the interval of the IO stream in the file, wherein, when other parameters remain unchanged, the greater the length of the IO stream in the file, the greater the length of the IO stream in the file. The greater the sequence degree of the IO stream, the stronger the sequence of the IO stream in the file; when other parameters remain unchanged, the larger the interval of the IO stream in the file, the smaller the sequence degree of the IO stream in the file, the stronger the sequence of the IO stream in the file. The less sequential the stream is.

(5) Operation

Based on the dimension of operation, the short-term access feature includes at least one of the following features: operation command proportion, host operation distribution, and operation mode distribution.

Wherein, the host operation refers to the user's operation on the host identified based on the batch IO of the file system service, and may also be referred to as aggregation operation, user operation, and the like. The host operation can be, for example, cp (indicates copying), rm (indicates deleting a directory or file), rmdir (indicating deleting an empty directory), grep (indicating query), cd (indicating switching), and ls (indicating viewing the current directory) on the Linux system. ), ll (indicates to view the current file), mkdir (indicates to create), mv (indicates to cut or rename), etc.

The operating mode distribution is used to indicate the quantity of each operating mode corresponding to the batch IO. Operation modes include but are not limited to sequential read, sequential write, random read, random write, create write, append write, overwrite, file lock, protocol lock, file system lock, etc.

The proportion of operation commands can be the proportion of IO operations, the proportion of host operations, and so on. Among them, the proportion of IO operations is the ratio of the number of IO operations in the batch IO (the corresponding fields are read read, write wirte, or look up, etc.) to the total number of IOs in the batch IO, and the proportion of host operations is the slave batch The ratio of the number of host operations identified by the IO to the total number of IOs in the batch IO.

In this embodiment of the application, in the file system business scenario, the first workload feature also includes a global feature, which is used to indicate the hierarchical structure distribution of the file system, and the global feature includes at least one of the following features: The number of files, the number of directories in the file system, the distribution of directory depth in the file system, the distribution of the number of files in the directory of the file system, the distribution of file access frequency in the file system, and the distribution of directory access frequency in the file system.

In some possible embodiments, the global feature may also include the distribution of the number of subdirectories at each directory depth, the distribution of the number of files at each directory depth, and the distribution of file sizes at each directory depth.

In a specific implementation, obtaining the first workload feature of the storage device may be: the processor of the storage device obtains the first workload feature of the storage device based on the workload data in the memory. It can be seen that the first workload feature is extracted online by the storage device, not by the external device offline. In this way, the impact of copying IO data by the external device on the performance of the storage device is effectively avoided, and the extraction of the workload feature is improved. efficiency.

Exemplarily, in the block business scenario, the workload data includes: the logical unit number (Logical Unit Number, LUN) of the IO access of the block business, the offset of the starting position of the IO, the size of the IO, and the type of the IO operation (for example, read operation or write operation) and other data.

Exemplarily, in the file system business scenario, the workload data includes: the identification ID of the file system, the client IP address corresponding to the IO, the file ID accessed by the IO, the directory ID accessed by the IO, and the start position offset of the IO , IO size, IO operation type (for example, read operation, write operation, or metadata operation), etc.

It can be seen that the workload features of the above-mentioned block business and the workload feature of the file system business include related features obtained based on the reuse distance, such as the reuse distance distribution characteristics of the block business, and the file reuse distance distribution and directory reuse distance of the file system business. Distribution, reuse distance distribution of address blocks within a file, etc.

Reuse distance is an important feature that can effectively characterize the distribution of IO hotspots. In the embodiment of the present application, based on the historical distribution information of repeated blocks in the block access sequence, the block access sequence is traversed once to obtain the reuse distance distribution of the block access sequence, which is beneficial to improve the extraction efficiency of the reuse distance distribution feature.

Referring to FIG. 4 , FIG. 4 is a schematic diagram of a block access sequence provided by an embodiment of the present application. Figure 4 shows the block access sequence {BCACDABCBCEA}. It can be seen that block B is accessed 3 times, block C is accessed 4 times, block A is accessed 3 times, block D is accessed 1 time, block E was visited 1 time. In order to accurately calculate the reuse distance of each pair of accesses, it is necessary to track the access information of each logical address block (hereinafter referred to as address block). Take a pair of accesses of A (that is, A ² and A ³ ) in the block access sequence as an example, where the superscript of A indicates the number of accesses to A:

Indicates the number of address blocks between the second visit to A (denoted as A ² ) and the third visit to A (denoted as A ³ ), And the block access sequence between A ² and A ³ is {BCBCE}, so

Indicates the number of partially repeated address blocks between the second access to A and the third access to A; based on the block access sequence between A ² and A ³ is {BCBCE}, only block B and block C are respectively is visited twice, i.e. a partial repetition of block B and a partial repetition of block C, so

Indicates the number of globally repeated address blocks between the second visit to A and the third visit to A; it can be seen that before A ² , block B and block C have been visited for the first time, and A ² and A ³ Between block B and block C are added two visits, so

Indicates the reuse distance between the second visit A and the third visit A; it should be noted that, based on the definition of reuse distance, it is easy to know

in, and The three satisfy the quantitative relationship that is known and when, available In order to improve the calculation efficiency of the reuse distance, when traversing each logical address block of the block access sequence at one time, the interval distance distribution information between repeated blocks, the total number of address blocks corresponding to the current address block, and the number of globally repeated address blocks can be recorded respectively. According to the interval distance distribution information between repeated blocks, the probability density function of the reuse distance is fitted. Based on the probability density function, the probability that any interval distance is repeated in the interval composed of repeated blocks can be estimated.

reuse distance As an example Obtaining process: According to the total number of address blocks corresponding to A ² and the total number of address blocks corresponding to A ³ , the above According to the number of globally repeated address blocks corresponding to A ² and the number of globally repeated address blocks corresponding to A ³ , the above According to the probability density function and get the above final basis and get

According to the above method, the distribution feature of the reuse distance of the block service can be extracted, and the above method is also suitable for extracting the feature related to the reuse distance of the file system service. For example, when extracting the feature of file reuse distance distribution, the block access sequence shown in Figure 4 can be replaced by file ID access sequence; when extracting the directory reuse distance distribution feature, the block access sequence shown in Figure 4 can be replaced by directory ID access sequence Sequence; when the reuse distance distribution feature of the address blocks in the file is extracted, each address block in the block access sequence shown in FIG. 4 is each address block in the same file.

S102: Store the first workload feature.

In this embodiment of the present application, the first workload feature may be stored in a memory of a storage device. It should be noted that the first workload feature extracted in a short period of time can be stored in the memory of the storage device, so that other modules of the storage device can directly obtain it from the memory when they need to use the corresponding workload feature, which is conducive to improving data security. Transmission rate.

In some possible embodiments, when the sum of the data amounts of the first workload characteristics of each period stored in the memory of the storage device reaches the upper threshold, the first workload characteristics of each period in the memory may also be transferred to the storage device. stored on the hard disk.

In some possible embodiments, the storage device may also generate a visualized load profile according to the first workload feature, where the load profile includes various load features and performance bottleneck information. Wherein, in the block business scenario, the load profile is the load profile of the block business; in the file system business scenario, the load profile is the load profile of the file system business. It should be noted that the load profile can be displayed at different granularities based on user requirements. For example, the granularity can be divided into storage devices, controllers, and LUNs.

Optionally, in some possible embodiments, it is also possible to execute:

S103: Perform memory allocation, data migration, NAS load balancing, hot and cold data block identification, prefetch policy optimization, performance bottleneck perception, load prediction or load change perception according to the first workload feature.

In the embodiment of the present application, the first workload feature can provide support for the optimization of multiple modules of the storage device, and the multiple modules can be, for example, a memory allocation module, a hot and cold data identification module, a NAS load regulation module, and the like. Among them, the memory allocation module can perform memory allocation, prefetch strategy optimization, etc., and the hot and cold data identification module can perform hot and cold data block identification, data Moving, etc., the NAS load control module can perform NAS load balancing, load forecasting, etc.

It should be noted that when different modules in the storage device perform tuning operations, they may rely on the same or different workload characteristics. In addition, the workload features that the module relies on when performing the optimization operation may be some or all of the first workload features, which are not specifically limited here.

Exemplarily, in the block business scenario, the memory allocation module mainly performs memory allocation according to the above-mentioned hotspot distribution characteristics and IO size distribution and other characteristics. Specifically, the memory allocation module can determine the interval distance corresponding to the hotspot IO in the batch IO based on the distribution characteristics of the hotspot, and obtain the size of the memory cache when the maximum memory cache hit rate is achieved based on the interval distance corresponding to the hotspot IO, thereby realizing memory Allocation of resources. It should be noted that the cache hit ratio refers to the ratio of the cache hits to the total cache accesses. The so-called cache hit means that the logical address to be read is located in the cache, and can be quickly read from the cache, which is called a cache hit.

Exemplarily, the prefetch policy optimization may be performed according to the proportion of sequence flow and interval flow, IO flow length distribution, IO flow bandwidth distribution and IO flow interval distribution in flow characteristics. For different LUNs in the block business, a unified prefetching strategy is often adopted, but when the overall prefetching has a large waste rate, the prefetching function of each LUN will be stopped. However, in the embodiment of this application, according to the characteristics of each LUN, such as the proportion of sequence flow and interval flow, IO flow length distribution, IO flow bandwidth distribution, and IO flow interval distribution, etc., the benefit of enabling the prefetch function for each LUN is evaluated. The revenue corresponding to each LUN dynamically adjusts the cache prefetch strategy to reduce the read amplification caused by the prefetch strategy as much as possible while ensuring the prefetch hit rate to prevent resource waste.

Exemplarily, the NAS load balancing module is used to implement load balancing of file system services. Specifically, according to the short-term access characteristics and global characteristics of the file system, the NAS load balancing module can pre-evaluate the data volume and access frequency of the directory or file when creating a directory or file, so as to determine the directory or file to be created. The handler, implemented to match the appropriate handler for newly created directories or files. In addition, when it is determined that there is a large difference in the workload of each processor, according to the short-term access characteristics and global characteristics of the file system business, the frequently accessed directories and files can be moved to relatively idle or less workload processors. device. In this way, NAS load balancing is realized.

Exemplarily, the historical access frequency and reuse distance distribution of each data block (or address block) may also be determined according to the hotspot distribution feature in the first workload feature, and the historical access frequency and reuse distance distribution of the data block are used for the Data blocks are identified as hot and cold data blocks.

In some possible embodiments, after determining the hot and cold attributes of the data block, data migration may also be performed. For example, in fusion storage, try to store hot data in high-performance solid-state disk SSD and place cold data in low-performance mechanical hard disk HDD, which can save the storage space of high-performance SSD. Therefore, after identifying data block A as For hot data blocks but data block A is placed on HDD, the data corresponding to data block A can be migrated to SSD to improve data reading efficiency.

In some possible embodiments, the storage device may also perform load prediction or load change sensing according to the extracted first workload feature, so as to estimate a future load change trend, so as to better implement load control. In some possible embodiments, the storage device may also perform performance bottleneck detection, stabilize write bandwidth, etc. according to the first workload feature, which is not specifically limited here.

It can be seen that, by implementing the embodiment of the present application, different workload characteristics are extracted from multiple dimensions for block business and file system business, which can more accurately characterize the workload of storage devices in different business scenarios. In addition, the online extraction of workload characteristics of storage devices not only improves the extraction efficiency of action load characteristics, but also effectively avoids the direct leakage of user IO data and improves the security of user data. The extracted workload characteristics can be directly used by each module of the storage device itself, so as to realize the self-adaptive optimization of the storage device and help to improve the intelligence of the storage system.

Referring to FIG. 5 , FIG. 5 is a flow chart of a workload feature extraction method provided by an embodiment of the present application, which is applied to a communication system composed of a storage device and an emulation device. The embodiment in Figure 5 can be a supplement to the embodiment in Figure 3, or it can be independent of Figure 3 Example. The method includes but is not limited to the following steps:

S201: The storage device acquires its own first workload characteristic during the IO execution process. For details of this step, reference may be made to the related description of S101 in the embodiment in FIG.

S202: The storage device sends the first workload feature to the emulation device. Correspondingly, the simulation device receives the first workload feature sent by the storage device.

S203: The simulation device performs memory allocation, data migration, NAS load balancing, hot and cold data block identification, prefetch policy optimization, performance bottleneck perception, load prediction or load change perception according to the first workload characteristics.

In this embodiment of the application, the simulation device can execute at least one of the following applications based on the first workload feature: memory allocation, data movement, network attached storage NAS load balancing, identification of hot and cold data blocks, prefetch strategy optimization, performance bottleneck sensing, load forecasting, or load change sensing. It should be noted that for the execution process of applications such as memory allocation, data migration, and NAS load balancing, please refer to the relevant description of S103 in the embodiment of FIG. 3 above.

In some possible embodiments, before the simulation device executes each of the above applications, the simulation device may first verify whether the extracted first workload feature is authentic, and then execute the above application if the first workload feature is determined to be authentic.

Exemplarily, the simulation device determines that the extracted first workload feature is credible, which may be: the simulation device simulates the first workload feature to obtain IO simulation data; re-extracts the third workload feature according to the IO simulation data, and compares the third workload feature With the first workload feature, when the third workload feature is consistent with the first workload feature, it is determined that the first workload feature is credible, which means that the IO simulation data can be used as the real IO data corresponding to the first workload feature. It can be understood that the first workload feature is credible, indicating that the first workload feature can better characterize the workload of the storage device, and the first workload feature and the IO simulation data obtained according to the first workload feature are of reference significance.

It should be noted that the consistency between the third workload feature and the first workload feature may be that the similarity between the third workload feature and the first workload feature satisfies a preset condition, for example, the third workload feature and the first workload feature The similarity is greater than or equal to a preset threshold.

Exemplarily, the simulation device determines that the extracted first workload feature is credible, which may be: the simulation device simulates the first workload feature to obtain IO simulation data; observe the performance index of the IO simulation data when running on the device to obtain the IO simulation The difference value between the performance index corresponding to the data and the performance index corresponding to the real workload data. When the difference value is less than the preset difference threshold, it is determined that the first workload feature is credible, and the IO simulation data can be used as the first workload feature corresponding real workload data. It should be noted that the performance indicators may be IOPS, IO delay, cache hit IO quantity, cache prefetch IO quantity, and so on.

In some possible embodiments, the simulation device can also assist performance fault location, offline testing, algorithm strategy evaluation, etc. based on the first workload feature.

Optionally, in order to increase the data transmission rate between the storage device and the simulation device, the storage device may also compress the first workload feature, and then send the compressed workload feature to the simulation device. In this case, the above S202 and S203 may not be executed, but the following S204-S206 are executed:

S204: The storage device compresses the first workload feature to obtain a second workload feature and compression parameters.

Wherein, the number of features included in the second workload feature is smaller than the number of features included in the first workload feature, and the compression parameters are used to restore the second workload feature to the first workload feature. It can be seen that compressing the first workload feature can not only reduce the consumption of storage space for storing the first workload feature, but also increase the data transmission rate.

It should be noted that the second workload feature is a compressed workload feature of the first workload feature. In other words, each feature in the second workload feature is a feature that is retained after the first workload feature is compressed. It can be understood that the number of features in the second workload feature is smaller than the number of features in the first workload feature, so the data volume of the second workload feature is also smaller than the data volume of the first workload feature.

In this embodiment of the present application, each feature in the second workload features corresponds to one or more features in the first workload features.

For example, if feature a1 in the second workload feature corresponds to feature a2 and feature a3 in the first workload feature, it means that feature a2 and feature a3 can be restored or recovered based on the compression parameters and feature a1.

In the embodiment of the present application, the field names of each feature in the second workload feature remain unchanged before and after compression, so that it can be ensured that each feature in the second workload feature is still interpretable.

For example, assume that the first workload feature includes feature 1, feature 2, and feature 3, and the second workload feature obtained after compressing the first workload feature only includes feature 1. In this case, feature 1 is the compressed first workload feature The feature that is reserved later, and the field name of feature 1 in the first workload feature is the same as the field name of feature 1 in the second workload feature.

In the implementation of the present application, compressing the first workload features includes: compressing the first workload features according to the similarity and/or predictability between features in the first workload features.

In a specific implementation, the similarity between features in the first workload feature may refer to the similarity between different features of the same batch of IOs in the first workload feature. In this case, the compression process may be: for any two different features in the first workload feature, calculate the similarity between the two features, when the similarity between the two features meets the preset similarity condition , delete any one of the two features corresponding to the similarity. It should be noted that before deleting any feature corresponding to the similarity, the mapping relationship between the two features corresponding to the similarity needs to be recorded in the compression parameters.

For example, take feature A and feature B in the first workload feature as an example, calculate the similarity between feature A and feature B, and when the similarity between feature A and feature B is greater than or equal to the preset similarity threshold, it can be considered If feature A is similar to feature B, any one of feature A and feature B can be deleted, thereby effectively reducing the data volume of the first workload feature, thereby realizing the compression of the first workload feature.

In another specific implementation, the similarity between features in the first workload feature may refer to the similarity between features of different batches of IOs in the first workload feature.

For example, assume that the first workload features include feature 1, feature 2, feature 3, feature 4, feature 5, and feature 6, where feature 1, feature 2, and feature 3 belong to the first batch of IOs, and feature 4, feature 5, and feature 6 belongs to the second batch of IO, the field name of feature 1 is the same as that of feature 4, the field name of feature 2 is the same as that of feature 5, the field name of feature 3 is the same as that of feature 6, and the field name of feature 1 The content of feature 4 is the same as that of feature 4 (that is, the similarity meets the preset similarity conditions), the content of feature 2 is the same as that of feature 5 (that is, the similarity meets the preset similarity conditions), but the content of feature 3 is the same as that of feature 6 are not the same (that is, the similarity does not meet the preset similarity conditions), in this case, the three features of the first batch of IOs are all retained, because feature 4 is the same as feature 1, feature 5 is the same as feature 2, and feature 1 and Feature 2 has been reserved, so redundant features 4 and 5 can be deleted, that is, only feature 6 is reserved for the second batch of IOs. In this way, the compression of the first workload feature is realized.

In the embodiment of the present application, compressing the first workload feature according to the predictability among the features in the first workload feature includes: determining the predictable feature in the first workload feature through the artificial intelligence model, and obtaining the artificial intelligence model parameters, and remove predictable features from workload features, where the parameters of the artificial intelligence model are included in the compressed parameters.

For example, assuming that the first workload feature includes feature b1, feature b2, feature b3, and feature b4, and the artificial intelligence model determines that feature b2 can be predicted based on feature b1, and feature b4 can be predicted based on feature b3, then it can be deleted from the first workload feature Feature b2 and feature b4, or, remove feature b1 and feature b3 from the first workload feature. In addition, the obtained compression parameters include conversion parameters between feature b1 and feature b2, and conversion parameters between feature b3 and feature b4.

Exemplarily, the artificial intelligence model is also a single-layer neural network, a random forest (Random Forest, RF) model, a support vector machine (Support Vector Machine, SVM) model or other prediction algorithms, which are not specifically limited herein.

It should be noted that the prediction of features can be one-to-one, that is, predict another feature based on one feature, or many-to-one, that is, predict a feature based on multiple features, or one-to-many, that is, based on one feature Multiple features are predicted, which is not specifically limited in this embodiment of the present application.

It can be understood that the larger the number of predictable features in the first workload feature, the higher the compression ratio of the first workload feature that can be achieved, and the larger the data volume of the second workload feature obtained after the first workload feature is compressed. Small.

In some possible embodiments, the features of the first workload may also be compressed in combination with the similarity and predictability between features. For details, please refer to the related descriptions about the similarity and predictability above, which will not be repeated here.

Optionally, in some possible embodiments, it is also possible to execute:

S205: The storage device sends the second workload characteristics and compression parameters to the emulation device.

Correspondingly, the simulation device receives the second workload feature and compression parameters sent by the storage device. In this way, transmitting the second workload characteristics and compression parameters to the simulation device can effectively improve the data transmission efficiency between the storage device and the simulation device.

S206: The simulation device performs memory allocation, data movement, NAS load balancing, identification of hot and cold data blocks, prefetch policy optimization, performance bottleneck perception, load prediction or load change perception according to the second workload characteristics and compression parameters.

In the embodiment of the present application, the simulation device can first use the compression parameters to restore the second workload feature to the first workload feature, and perform the above-mentioned memory allocation, data movement, NAS load balancing, hot and cold data block identification, etc. according to the first workload feature. Prefetching strategy tuning, performance bottleneck perception, load forecasting or load change perception. It should be noted that for details about memory allocation, data migration, and NAS load balancing performed by the simulation device according to the characteristics of the first workload, please refer to the relevant description of S203 above. For the sake of brevity, details are omitted here.

It can be seen that, implementing the embodiment of the present application, the storage device extracts different workload characteristics for the block service and the file system service respectively, which can more accurately characterize the load characteristics of the storage device in different service scenarios. Compressing the extracted workload characteristics and then transmitting them to the simulation device can effectively improve the data transmission efficiency between the storage device and the simulation device, and also enable the simulation device to realize the reproduction of business scenarios based on the workload characteristics.

Referring to FIG. 6 , FIG. 6 is a schematic diagram of a functional structure of a storage device provided by an embodiment of the present application. The storage device 30 includes a processing unit 310 and a storage unit 312 . The storage device 30 may be implemented by hardware, software, or a combination of software and hardware.

Wherein, the processing unit 310 is configured to obtain the first workload feature of the storage device during the execution of the input and output IO; the storage unit 312 is used to store the first workload feature, and the first workload feature is used for memory allocation of the storage device , data migration, network attached storage NAS load balancing, identification of hot and cold data blocks, prefetch policy tuning, performance bottleneck perception, load forecasting or load change perception.

In some possible embodiments, the storage device 30 further includes a sending unit 314, and the sending unit 314 is configured to send the second workload characteristics and compression parameters to the simulation device, wherein the compression parameters are used to restore the second workload characteristics to the first workload feature. In some possible embodiments, the sending unit 314 may also be configured to send the first workload feature to the simulation device.

Each functional module of the storage device 30 may be used to implement the method described in the embodiment of FIG. 3 . In the embodiment in FIG. 3 , the processing unit 310 may be used to execute S101 and S103 , the storage unit 312 may be used to execute S102 , and the sending unit 314 may be used to execute S202 or S205 in FIG. 5 . Each functional module of the storage device 30 can also be used to implement the method on the storage device side described in the embodiment of FIG. 5 , and details are not repeated here for the sake of brevity.

One or more of each unit in the above embodiment shown in FIG. 6 may be realized by software, hardware, firmware or a combination thereof. The software or firmware includes but is not limited to computer program instructions or codes, and can be executed by a hardware processor. The hardware includes but not limited to various integrated circuits, such as central processing unit (central processing unit, CPU), digital signal processor (digital signal processor, DSP), field-programmable gate array (field-programmable gate array, FPGA) Or application-specific integrated circuit (ASIC).

The present application also provides a storage device. As shown in FIG. 7 , the storage device 40 includes: a processor 401 , a communication interface 402 , a memory 403 and a bus 404 . The processor 401 , the memory 403 and the communication interface 402 communicate through a bus 404 . Storage device 40 may be a server or a storage device. It should be understood that the present application does not limit the number of processors and memories in the storage device 40 .

The bus 404 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one line is used in FIG. 7 , but it does not mean that there is only one bus or one type of bus. The bus 404 may include a pathway for transferring information between various components of the storage device 40 (eg, memory 403 , processor 401 , communication interface 402 ).

The processor 401 may include any one or more of processors such as a central processing unit (central processing unit, CPU), a microprocessor (micro processor, MP), or a digital signal processor (digital signal processor, DSP).

The memory 403 is used to provide a storage space, in which data such as operating systems and computer programs can be stored. Memory 403 can be random access memory (random access memory, RAM), erasable programmable read only memory (erasable programmable read only memory, EPROM), read-only memory (read-only memory, ROM), or portable read-only memory One or more combinations of memory (compact disc read memory, CD-ROM), etc. The memory 403 may exist independently, or may be integrated inside the processor 401 .

Communication interface 402 may be used to provide information input or output to processor 401 . Or alternatively, the communication interface 402 can be used to receive data sent from the outside and/or send data to the outside, and can be a wired link interface such as an Ethernet cable, or a wireless link (such as Wi-Fi, Bluetooth, general wireless transmission, etc.) interface. Or alternatively, the communication interface 402 may further include a transmitter (such as a radio frequency transmitter, an antenna, etc.) or a receiver coupled with the interface.

The processor 401 in the storage device 40 is configured to read the computer program stored in the memory 403 to execute the aforementioned method, such as the method described in FIG. 3 or the method on the storage device side described in FIG. 5 .

In a possible design manner, the storage device 40 may be one or more modules in the execution subject of the method shown in FIG. 3 , and the processor 401 may be used to read one or more computer programs stored in the memory, Used to do the following:

During the execution of the input and output IO, acquire the first workload feature of the storage device;

The first workload feature is stored in the storage unit 312, and the first workload feature is used for memory allocation of storage devices, data migration, network attached storage NAS load balancing, hot and cold data block identification, prefetch policy optimization, performance bottleneck perception, and load prediction or load change sensing.

In the above-mentioned embodiments herein, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

It should be noted that those skilled in the art can see that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium , storage medium includes read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), programmable read-only memory (Programmable Read-only Memory, PROM), erasable programmable read-only memory ( Erasable Programmable Read Only Memory, EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory, EEPROM, Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, tape storage, or any other computer-readable medium that can be used to carry or store data.

The essence of the technical solution of the present application or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of software products. The computer program product is stored in a storage medium, including several instructions. So that a device (which may be a personal computer, a server, or a network device, a robot, a single-chip microcomputer, a chip, a robot, etc.) executes all or part of the steps of the methods described in the various embodiments of the present application.

Claims

A workload feature extraction method applied to a storage device, wherein the method comprises:

During the execution of the input and output IO, acquire the first workload feature of the storage device;

Storing the first workload feature, the first workload feature is used for memory allocation, data migration, network attached storage NAS load balancing, hot and cold data block identification, prefetch policy optimization, performance bottleneck perception, Load forecasting or load change sensing.
The method according to claim 1, wherein the acquiring the first workload feature of the storage device comprises:

The processor of the storage device obtains the first workload feature according to the workload data in the memory.
The method according to claim 2, wherein the first workload feature is extracted offline without external equipment.
The method according to any one of claims 1-3, wherein, in a block business scenario, the first workload feature includes at least one of a time feature, a flow feature, and a hotspot distribution feature, wherein the The time feature is used to indicate the time interval corresponding to the IO of the block service, the flow feature is used to indicate the access mode of the IO flow of the block service, and the hot spot distribution feature is used to indicate the address block in the block service Reuse the distance distribution.
The method according to claim 4, wherein the first workload feature further includes at least one of the following features: IO size distribution, read-write ratio parameter, total read-write bandwidth, and the number of unaligned IOs.
The method according to any one of claims 1-3, wherein in the file system service scenario, the first workload feature includes a short-term access feature, and the short-time access feature is used to indicate the file system The access mode of business batch IO in at least one dimension of files, directories, time, IOs in files, and operations.
The method according to claim 6, wherein the first workload feature further includes a global feature, the global feature is used to indicate the hierarchical structure distribution of the file system, and the global feature includes at least one of the following features :

the number of files in the file system;

the number of directories in the file system;

The directory depth distribution of the file system;

The number distribution of files under the directory of the file system;

a file access frequency distribution of the file system; and

Directory access frequency distribution of the file system.
The method according to any one of claims 1-7, wherein the method further comprises:

Compressing the first workload features to obtain second workload features and compression parameters, the number of features included in the second workload features is smaller than the number of features included in the first workload features, and the compression parameters are used to compress The second workload characteristic reverts to the first workload characteristic;

According to the second workload feature and the compression parameters, perform NAS load balancing, memory allocation, data migration, identification of hot and cold data blocks, prefetch policy optimization, performance bottleneck perception, load prediction or load change perception.
The method according to claim 8, characterized in that the method further comprises:

Sending the second workload feature and the compression parameter to the simulated device, so that the simulated device performs memory allocation, data movement, network attached storage NAS load balancing, heating and cooling according to the second workload feature and the compression parameter Data block identification, prefetch strategy tuning, performance bottleneck perception, load prediction or load change perception.
The method according to claim 9, wherein the second workload feature and the compression parameters are also used for the simulation device to obtain IO simulation data and verify the first workload feature based on the IO simulation data Is it credible.
A workload feature extraction device, characterized in that the device comprises:

a processing unit, configured to acquire the first workload feature of the storage device during the execution of the input and output IO;

The storage unit is configured to store the first workload feature, and the first workload feature is used for memory allocation, data migration, network attached storage NAS load balancing, hot and cold data block identification, and prefetch policy optimization of the storage device , performance bottleneck perception, load forecasting or load change perception.
The device according to claim 11, wherein the processing unit is specifically used for:

The first workload feature is obtained by the processor according to the workload data in the memory.
The device according to claim 12, wherein the feature of the first workload is extracted offline without external equipment.
The device according to any one of claims 11-13, wherein, in a block service scenario, the first workload feature includes at least one of a time feature, a flow feature, and a hotspot distribution feature, wherein the The time feature is used to indicate the time interval corresponding to the IO of the block service, the flow feature is used to indicate the access mode of the IO flow of the block service, and the hot spot distribution feature is used to indicate the address block in the block service Reuse the distance distribution.
The device according to claim 14, wherein the first workload feature further includes at least one of the following features: IO size distribution, read-write ratio parameter, total read-write bandwidth, and the number of unaligned IOs.
The device according to any one of claims 11-13, wherein in the file system business scenario, the first workload feature includes a short-term access feature, and the short-time access feature is used to indicate the file system The access mode of business batch IO in at least one dimension of files, directories, time, IOs in files, and operations.
The device according to claim 16, wherein the first workload feature further includes a global feature, the global feature is used to indicate the hierarchical structure distribution of the file system, and the global feature includes at least one of the following features :

the number of files in the file system;

the number of directories in the file system;

The directory depth distribution of the file system;

The number distribution of files under the directory of the file system;

a file access frequency distribution of the file system; and

Directory access frequency distribution of the file system.
The device according to any one of claims 11-17, wherein the processing unit is further configured to:

Compressing the first workload features to obtain second workload features and compression parameters, the number of features included in the second workload features is smaller than the number of features included in the first workload features, and the compression parameters are used to compress The second workload characteristic reverts to the first workload characteristic;

According to the second workload feature and the compression parameters, perform NAS load balancing, memory allocation or data migration, identification of hot and cold data blocks, prefetch policy optimization, performance bottleneck perception, load prediction or load change perception.
The device according to claim 18, further comprising:

A sending unit, configured to send the second workload characteristics and the compression parameters to the simulation device, so that the simulation device performs memory allocation, data movement, and network attached storage (NAS) according to the second workload characteristics and the compression parameters Load balancing, identification of hot and cold data blocks, prefetch strategy tuning, performance bottleneck perception, load forecasting or load change perception.
The apparatus according to claim 19, wherein the second workload feature and the compression parameter are also used by the simulation device to verify whether the first workload feature is credible.
A device, characterized in that the device comprises a memory and a processor, and the memory is used to store program instructions; when the processor executes the program instructions in the memory, the computer performs the process described in claim 1. - the method described in any one of 10.
A computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and the program instructions are used to implement the method according to any one of claims 1-10.