WO2023115532A1

WO2023115532A1 - Data processing method and data processing apparatus

Info

Publication number: WO2023115532A1
Application number: PCT/CN2021/141179
Authority: WO
Inventors: 底晔佳; 贾学超; 冉宜; 黄恩走; 玄在雄
Original assignee: 华为技术有限公司
Priority date: 2021-12-24
Filing date: 2021-12-24
Publication date: 2023-06-29
Also published as: CN116830074A

Abstract

A data processing method and a data processing apparatus. According to the method, a storage device (110) obtains operation parameters of a plurality of read-write operations executed within a period of time, determines a performance specification of the storage device (110) on the basis of these operation parameters, and sends the performance specification to a task allocation device (102) outside a storage system (101). In this way, the storage device (110) can determine the performance specification on the basis of the read-write operations in an actual scene instead of determining on the basis of a test environment, so that a more accurate performance specification can be obtained.

Description

Data processing method and data processing device

technical field

Embodiments of the present disclosure mainly relate to the chip field, and more specifically, relate to a data processing method and a data processing device.

Background technique

In recent years, storage devices such as Solid State Disk (SSD) have been widely used in various storage scenarios, such as servers, data centers, embedded equipment, etc. Since the development of SSD, various models for SSD performance evaluation have been proposed. The usual practice is to treat SSD as a black box, and establish a model for SSD performance evaluation through the performance of multiple test scenarios. However, since the test scenario is different from the actual scenario, the accuracy of the evaluated SSD performance may be low, and there is a large deviation from the actual situation.

Contents of the invention

Embodiments of the present disclosure provide a data processing method and a data processing device, which can determine more accurate performance specifications based on read and write operations in actual scenarios.

In a first aspect of the present disclosure, a data processing method is provided. The method includes: the storage device acquires operating parameters of multiple read and write operations performed within a period of time; the storage device determines performance specifications of the storage device based on the operating parameters; and the storage device sends the performance specifications to a task allocation device outside the storage system.

In this way, the storage device can determine the performance specification based on the operating parameters of the read and write operations in the actual scene, rather than based on the test environment, so that the obtained performance specification is more accurate.

In a possible implementation manner, the determining the performance specification by the storage device based on the operating parameters includes that the storage device determines the performance specification based on the operating parameters in response to receiving a request for performance of the storage device from the task allocation device.

In this way, the performance specification can be determined based on the performance request of the storage device from the task allocation device, and the task allocation device can further allocate read and write operations based on the performance specification from the storage device, so as to make full use of the performance of the storage device and realize data processing. Layout optimization to improve overall performance. Moreover, the embodiments of the present disclosure can realize the interaction between the storage device and the task distribution device.

In a possible implementation manner, the determination of the performance specification by the storage device based on the operating parameters includes if the number of multiple read and write operations exceeds the number threshold, or if the current time point is different from the last time the storage device sent the performance specification to the task allocation device The length of time between points exceeds a time threshold, and the storage device determines performance specifications based on operating parameters.

In this way, the performance specification can be determined based on the preset times threshold or time threshold, and then the performance specification can be dynamically provided to the task management device. Therefore, the task management device can periodically obtain the performance specification based on the number of times threshold or the event threshold, which saves the signaling overhead of the task management device sending a request to obtain the performance specification.

In a possible implementation manner, the storage device determining the performance specification based on the operating parameters includes that the storage device determines the node performance specifications of each node in the storage device based on the operating parameters, and the nodes include at least one of an interface, a channel, a flash memory particle, and a controller chip. item; and the storage device determines the performance specification based on the node performance specification of each node.

In this way, the node performance of each node on the data transmission path can be considered. Since the architecture of the storage device includes nodes such as interfaces, channels, flash memory particles, and controller chips, the architecture of the storage device is fully integrated by considering the node performance. Get more accurate storage device performance specifications.

In a possible implementation manner, the storage device determining the performance specification based on the operating parameters includes the storage device inputting the operating parameters into a trained performance model to obtain the performance specification.

In this way, using a pre-trained performance model to determine performance specifications can make full use of existing data and improve the efficiency of determining performance specifications.

In a possible implementation manner, the running parameters include at least one of the following: read and write characteristic parameters, disk state parameters, or algorithm state parameters.

In this way, in the embodiments of the present disclosure, performance specifications are determined from multi-dimensional influencing factors such as read and write characteristic parameters, disk status, and algorithm status. On the one hand, the determined performance specifications are more accurate and close to the real situation; on the other hand, due to consideration Operating parameters in multiple dimensions so that the determined performance specification can indicate the behavioral characteristics of the SSD. Furthermore, the system design can be carried out by the task management device in cooperation with the performance specifications of multiple storage devices, so as to fully utilize the performance of the disk from a global perspective.

In a possible implementation manner, the characteristic parameters of reading and writing include at least one of the following: the number of reading and writing, the size of reading and writing, the heat of reading and writing, or the randomness of reading and writing; the parameters of the disk status include parameters about at least one of the following : Read and write concurrency, write amplification factor, space utilization, controller gear, or media specifications of flash memory particles; parameters of the algorithm status include indication information of whether the garbage collection GC algorithm is enabled, and/or redundancy of independent disks Array (Redundant Array of Independent Disks, RAID) configuration.

In this way, since the determination of parameters such as the write amplification factor takes into account the influence of algorithm changes inside the storage device, the performance specifications determined based on the operating parameters can be made more accurate.

In a possible implementation manner, the performance specification includes at least one of a steady-state performance specification and a maximum performance specification.

In a possible implementation, the steady-state performance specification includes at least one of the following: steady-state read IOPS, steady-state read bandwidth, steady-state write IOPS, or steady-state write bandwidth, and the maximum performance specification includes at least one of the following : Maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.

In a second aspect of the present disclosure, a data processing method is provided. The method includes: the task allocation device receives performance specifications from at least one storage device in the storage system; and the task allocation device allocates read and write operations to the at least one storage device based at least on the performance specification of the at least one storage device.

In this way, the task allocation device can allocate read and write operations based on performance specifications from the storage device, and can make full use of the performance of the storage device to optimize data layout and improve overall performance.

In a possible implementation manner, the task allocation device further includes sending a request for performance of the at least one storage device to the at least one storage device.

In a possible implementation manner, the steady-state performance specification includes at least one of the following: steady-state read input/output times per second (Input/Output per Second, IOPS), steady-state read bandwidth, steady-state write IOPS, or steady-state The write bandwidth, and the maximum performance specification includes at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.

In a third aspect of the present disclosure, there is provided a data processing device, which includes: a first determination module configured to obtain operating parameters of multiple read and write operations performed within a period of time; a second determination module configured to configured to determine the performance specification of the storage device based on the operating parameters; and a sending module configured to send the performance specification to a task allocation device outside the storage system.

In a possible implementation manner, the second determining module is configured to determine the performance specification based on the operating parameter in response to receiving a request for the performance of the storage device from the task allocation device.

In a possible implementation manner, the second determining module is configured to: if the number of multiple read and write operations exceeds the number threshold, or if the current time point is between the time point when the storage device sent the performance specification to the task allocation device last time A performance specification is determined based on the operating parameters for the length of time between which the time threshold is exceeded.

In a possible implementation manner, the second determination module is configured to determine the node performance specification of each node in the storage device based on the operating parameters, and the node includes at least one of an interface, a channel, a flash memory particle, and a controller chip; and based on The node performance specification of each node determines the performance specification.

In a possible implementation manner, the second determination module is configured to input the operating parameters into the trained performance model to obtain the performance specification.

In a possible implementation manner, the running parameters include at least one of the following: read and write characteristic parameters, disk state parameters, or algorithm state parameters. Optionally, the read and write characteristic parameters include at least one of the following: read and write times, read and write size, read and write heat, or read and write randomness; the parameters of the disk state include parameters about at least one of the following: read and write Concurrency, write amplification factor, space utilization, controller gear, or media specifications of flash memory particles; the parameters of the algorithm state include indication information of whether the garbage collection GC algorithm is enabled, and/or RAID configuration.

In a possible implementation manner, the performance specification includes at least one of a steady-state performance specification and a maximum performance specification. The steady-state performance specification includes at least one of the following: steady-state read IOPS, steady-state read bandwidth, steady-state write IOPS, or steady-state write bandwidth. The maximum performance specification includes at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.

In a fourth aspect of the present disclosure, there is provided a data processing apparatus, which includes a receiving module configured to receive performance specifications from at least one storage device in a storage system; and an allocation module configured to at least based on at least one storage device A performance specification for a device, distributing read and write operations to at least one storage device.

In a possible implementation manner, the apparatus further includes a sending module configured to send a request for performance of the at least one storage device to the at least one storage device.

In a fifth aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory, and the memory stores instructions executed by the processor. When the instructions are executed by the processor, the electronic device realizes the above-mentioned first aspect or any embodiment thereof, or the above-mentioned second aspect or The method in any of its embodiments.

In a sixth aspect of the present disclosure, a computer readable storage medium is provided. Computer-executable instructions are stored on the computer-readable storage medium. When the computer-executable instructions are executed by a processor, the above-mentioned first aspect or any embodiment thereof, or the above-mentioned second aspect or any embodiment thereof can be implemented. method.

In a seventh aspect of the present disclosure, a computer program or computer program product is provided. The computer program or computer program product is tangibly stored on a computer-readable medium and includes computer-executable instructions which, when executed, implement the computer program according to the above-mentioned first aspect or any embodiment thereof, or the above-mentioned second aspect. or the method in any of its embodiments.

In an eighth aspect of the present disclosure, a chip or chip system is provided. The chip or chip system includes a processing circuit configured to perform operations according to the method in the above first aspect or any embodiment thereof, or the above second aspect or any embodiment thereof.

It should be understood that what is described in the Summary of the Invention is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, identical or similar reference numerals denote identical or similar elements, wherein:

FIG. 1 shows a schematic diagram of an example scenario in which some embodiments of the present disclosure can be implemented;

Fig. 2 shows a schematic interaction diagram of a data processing process according to some embodiments of the present disclosure;

Fig. 3 shows a schematic block diagram of a data processing device according to some embodiments of the present disclosure;

FIG. 4 shows a schematic block diagram of a data processing apparatus implemented in a storage device as in FIG. 1 according to some embodiments of the present disclosure;

Fig. 5 shows a schematic block diagram of a data processing device according to some embodiments of the present disclosure; and

Fig. 6 shows a schematic block diagram of an example device that may be used to implement embodiments of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

In the description of the embodiments of the present disclosure, the term "comprising" and its similar expressions should be interpreted as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be read as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same object. The term "and/or" means at least one of the two items associated with it. For example "A and/or B" means A, B, or A and B. Other definitions, both express and implied, may also be included below.

Due to the characteristics of solid state disk (Solid State Disk, SSD) compared with hard disk drive (Hard Disk Drive, HDD), such as low access delay, high random performance, and high stability, SSD has been widely used in various storage scenarios, such as servers, Data centers, embedded devices, and more. Generally, a storage system includes multiple SSDs (also called SSD disks), and each SSD includes multiple flash memory particles. Based on this architecture, when the storage system receives IO requests from users, it will distribute read and write operations to multiple SSDs for execution. The speed of processing IO requests is a direct reflection of the overall performance of the storage system. The direct way to improve the overall performance is to allocate read and write operations according to the performance of SSD disks, so as to fully utilize the performance of each SSD disk. In order to achieve this goal, it is generally necessary to accurately evaluate the performance of SSD disks.

Usually, SSD manufacturers will obtain its steady-state performance in the test environment, and then use it as the SSD disk performance in the SSD product manual. But in fact, the performance of SSD disks is not fixed. On the contrary, it is changeable. Changeable means that there are many factors affecting the performance of SSD disks. From this perspective, the test environment is different from the actual scene. , so the steady-state performance obtained in the test environment cannot be equal to the performance in the actual scene; variable means that the performance of the SSD disk will change due to the change of the algorithm in the SSD. Therefore, a reliable solution capable of accurately determining the performance of SSD disks is needed.

In view of this, the embodiments of the present disclosure provide a solution for determining performance of SSD disks based on actual scenarios. According to the solution, the storage device determines corresponding operating parameters of multiple read and write operations within a period of time, determines performance specifications of the storage device based on these operating parameters, and sends the performance specifications to an external task allocation device. In this way, the storage device can determine the performance specification based on the read and write operations in the actual scene, so that the performance of the SSD disk can be obtained in a more accurate manner.

FIG. 1 shows a schematic diagram of an example scenario 100 in which some embodiments of the present disclosure may be implemented. In an example scenario 100, a storage system 101 and a task allocation device 102 are shown, where the storage system 101 includes storage devices 110-1 to 110-M, where M is a positive integer. To simplify the description, the multiple storage devices 110 - 1 to 110 -M may be collectively referred to as the storage device 110 .

Exemplarily, the task allocation device 102 may be configured to allocate read and write operations. Specifically, the task allocation device 102 may determine an IO queue based on a user's input/output (IO) request, and then allocate corresponding read and write operations in the IO queue to at least one storage device 110 . The storage device 110 may be configured to process read and write operations allocated by the task allocation device 102 .

As shown in FIG. 1 by taking the storage device 110-1 as an example, the storage device 110 may include an interface logic 112, a controller 114, a plurality of channels 118-1 to 118-N, and a -N corresponds to a plurality of flash memory particle groups 116-1 to 116-N, where N is a positive integer. To simplify the description, the plurality of flash particle groups 116 - 1 to 116 -N may be collectively referred to as the flash particle group 116 , and the plurality of channels 118 - 1 to 118 -N may be collectively referred to as the channel 118 . The flash memory die group 116 may include one or more flash memory dies (Die). It can be understood that multiple channels 118 can be concurrent, and multiple flash memory particles in the flash memory particle group 116 can also be concurrent.

The storage device 110 may be interconnected with the task distribution device 102 via the interface logic 112 . The channel 118 is a bridge between the controller 114 and the flash memory particle 116 , and multiple channels 118 can transmit data concurrently.

In some embodiments of the present disclosure, the storage device 110 may be implemented as an SSD, and the storage device 110 may also be called an SSD disk or other names. The task allocation device 102 may include a processor, but the implementation of the processor is not limited in the present disclosure, for example, it may include a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a neural network processing unit (Neural Network Processing Unit, NPU), or Digital Signal Processor (Digital Signal Processor, DSP), etc. For the convenience of description, the task distribution device 102 is referred to as a host (Host) hereinafter, but it should be understood that this name is only for illustration and should not be construed as a limitation of the present disclosure. Read and write operations may be referred to as IO operations, etc. This disclosure is not limited thereto.

It should be understood that the architecture of the storage device 110 shown in FIG. Each flash memory particle group 116 can be realized as NAND flash memory (NAND Flash) 117, or for example, the controller 114 can be referred to as a controller chip. Embodiments of the present disclosure are not limited to this.

The data processing solution in the embodiment of the present disclosure will be described below in conjunction with the scenario 100 described in FIG. 1 , taking the storage device 110 as an SSD and the task allocation device 102 as a host as an example. The solutions of the embodiments of the present disclosure can determine the disk performance of the SSD, where the disk performance can be characterized by the performance specification of the SSD.

FIG. 2 shows a schematic interaction diagram of a data processing process 200 according to some embodiments of the present disclosure. Process 200 involves SSD 210 and host 220. It can be understood that SSD 210 is an example of storage device 110 as shown in FIG. 1 , and host 220 is an example of task allocation device 102 as shown in FIG. 1 .

In the process 200, the SSD 210 obtains 201 operating parameters of multiple read and write operations performed within a period of time.

Specifically, in an actual scenario, the host 220 can allocate read and write operations based on IO requests from users. For example, the read and write operations can be: the host 220 reads data from the SSD 210 or writes data to the SSD 210. Correspondingly, the SSD 210 can obtain operating parameters associated with read and write operations.

The period of time may be from the last time SSD 210 sent performance specifications to host 220. It can be understood that the read and write operations performed within a period of time may include at least one read operation and/or at least one write operation. Then the SSD 210 can determine corresponding operating parameters for each read and write operation.

Exemplarily, the running parameter may be determined based on the statistics of multiple read and write operations, for example, to characterize the running state of the SSD 210 . In some embodiments of the present disclosure, the running parameters may include at least one of the following: read and write characteristic parameters, disk state parameters, or algorithm state parameters.

Exemplarily, the read/write characteristic parameter may include at least one item of read/write times, read/write size (size), read/write heat, and read/write randomness. Optionally, the number of reads and writes can represent the total number of multiple read and write operations, the read and write size can represent the total or average size of multiple read and write operations, and the read and write heat can represent the total or average heat of multiple read and write operations Heat, read and write randomness can represent the total randomness or average randomness of multiple read and write operations.

Specifically, the read and write characteristic parameters may include read parameters and write parameters. The read parameters may include at least one of the following: read times, read size, read heat, read randomness, and the like. The write parameters may include at least one of the following: write times, write size, write heat, write randomness, and the like.

Exemplarily, the host 220 may send a read and write command to the SSD 210, and the read and write command includes at least one of the following: a read and write type, a start address of a Logic Block Address (Logic Block Address, LBA), a length of the LBA, and the like. In some examples, the read/write type in the read/write command is "read", then SSD 210 can determine the read parameters. In some examples, the read/write type in the read/write command is "write", then SSD 100 can determine the write parameters.

Exemplarily, the status of the disc may include at least one of read and write concurrency, write amplification factor (Write Amplification Factor, WAF), space utilization, controller gear, media specifications of flash memory particles, and the like. Optionally, the read/write concurrency may represent the read/write concurrency of nodes in the disk that can be concurrent, for example, may include the read/write concurrency of channels and/or the read/write concurrency of flash memory particles. In some examples, the channel read concurrency may indicate the number of concurrent channels during read operations, and the channel write concurrency indicates the number of concurrent channels during write operations. In some examples, the read concurrency of flash memory particles may indicate the number of concurrent flash memory particles during a read operation, and the write concurrency of flash memory particles may indicate the number of concurrent flash memory particles during a write operation. For example, the space utilization rate can be expressed as a percentage, for example, the controller gear can be expressed as a specific gear value (such as 1, etc.), for example, the medium specification of the flash memory particle can be expressed as the access delay of the flash memory particle in response to read and write operations, etc. .

Exemplarily, the algorithm state may include whether the garbage collection (Garbage Collection, GC) algorithm is enabled, and correspondingly, the parameter of the algorithm state includes indication information whether the GC algorithm is enabled. Exemplarily, the algorithm state may include RAID configuration and the like. For example, the RAID configuration may include the maximum concurrent capability of the flash memory granules, and the like.

Embodiments of the present disclosure do not limit the specific manner in which the SSD 210 determines the operating parameters, for example, it may be implemented by means of counting or the like. In an embodiment where the operating parameters include WAF, SSD 210 may determine the WAF based on garbage collection (Garbage Collection, GC) triggered during the write operation. Specifically, SSD 210 receives a write command from host 220. During the writing process, it can determine whether to trigger a GC operation based on disk resources, and can determine the WAF by counting the number of GC operations triggered during the writing process. . Disk resources may include blank block resources in SSD hard disk space, referred to as blank space for short. Optionally, a GC operation may be triggered when the empty space is lower than a preset resource threshold.

In this way, the embodiment of the present disclosure also considers the influence of the change (ie whether it is enabled or not) of the internal algorithm (such as the GC algorithm) of the SSD on the operating parameters, so that more various performance specifications can be obtained.

It can be understood that the above-described operating parameters and the like are only illustrative, and in practical applications, the operating parameters may also include other parameters not listed above. Optionally, the above-mentioned terms are only illustrative, and in practical applications, the above-mentioned terms can be replaced by other names, for example, the number of reads and writes can be called the number of reads and writes, for example, the reading and writing heat can be called the number of reads and writes The temperature, such as reading and writing, may be referred to as IO, which is not limited in the embodiments of the present disclosure.

In process 200, SSD 210 determines 202 performance specifications for SSD 210 based on the operating parameters.

It can be understood that performance specifications may also be referred to as storage device performance specifications, storage performance specifications, SSD performance specifications, disk performance specifications, etc., or may also be referred to as storage device performance parameters or SSD performance parameters, etc., which are not discussed in this disclosure. limited.

In embodiments of the present disclosure, performance specifications may include steady-state performance specifications and/or maximum performance specifications. Optionally, the steady-state performance specification may be a performance specification when GC is enabled, and the maximum performance specification may be a performance specification when GC is not enabled.

In some embodiments, SSD 210 may determine performance specifications based on performance requests from host 220. As shown in FIG. 2, the host 220 may send 212 a performance request to the SSD 210. The performance request may be a request for the performance of the SSD 210. Accordingly, in response to receiving a request from host 220 for the performance of SSD 210, SSD 210 may determine performance specifications based on operating parameters.

In some examples, the capability request may be sent through a first specific field. For example, a first specific field may be predefined for the capability request. In some examples, the first flag field may be used to indicate that the next transmission is a capability request, and the capability request is sent immediately after the first flag field. In this way, the embodiment of the present disclosure can ensure that the performance request sent by the host 220 is correctly recognized by the SSD 210 by customizing the interface field, and can ensure correct transmission of the performance request by defining a recognizable language.

Optionally, if the performance request indicates a request for a steady-state performance specification, SSD 210 may determine a steady-state performance specification based on the performance request. Optionally, if the performance request indicates a request for a maximum performance specification, SSD 210 may determine the maximum performance specification based on the performance request. Optionally, if the performance request indicates a request for a steady-state performance specification and a maximum performance specification, SSD 210 may determine a steady-state performance specification and a maximum performance specification based on the performance request.

In some embodiments, SSD 210 may determine performance metrics based on event triggers. For example, the event trigger may include that the total number of multiple read and write operations performed within a period of time exceeds a number threshold, where the total number of read and write operations may be the sum of the number of reads and the number of writes.

In some embodiments, SSD 210 may periodically determine performance specifications. For example, the period may include a period of time to reach a time threshold. Or the period may include that the elapsed time from the last time the performance specification was sent to the host 220 exceeds the time threshold, that is, the time period between the time when the SSD 210 sent the performance specification to the host 220 last time and the current time exceeds the time threshold.

The performance specification determined by the SSD 210 may at least include Input/Output per Second (IOPS) and/or bandwidth. Specifically, the performance specification may include at least one of the following: read IOPS, read bandwidth, write IOPS, and write bandwidth. Correspondingly, it can be understood that the steady-state performance specification may include at least one of the following: steady-state read IOPS, steady-state read bandwidth, steady-state write IOPS, and steady-state write bandwidth. The maximum performance specification may include at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, and maximum write bandwidth.

In some embodiments, the SSD 210 can determine the node performance specification of each node in the multiple nodes of the SSD 210 based on the operating parameters, and further determine the SSD performance specification based on the node performance specification of each node. Multiple nodes may include interfaces, channels, flash particles, and controller chips through which data flows during the execution of read and write operations. And optionally, the SSD 210 may use the minimum node performance specification as the SSD performance specification.

Taking the maximum write IOPS as an example, the WAF can be close to 1 by stopping the GC operation on the SSD 210 architecture.

Taking steady-state write IOPS as an example, SSD 210 can determine interface steady-state write IOPS, channel steady-state write IOPS, and flash particle steady-state write IOPS, and then determine interface steady-state write IOPS, channel steady-state write IOPS, and flash particle steady-state The minimum value in the write IOPS is determined as the steady-state write IOPS of the SSD 210 .

Exemplarily, the node includes an interface, and the SSD 210 may determine the performance specification of the interface based on at least one of the following: the transmission rate of the interface, or the RAID configuration.

Exemplarily, the node includes a channel, and the SSD 210 can determine the performance specification of the channel based on at least one of the following: channel read-write concurrency, WAF, flash memory particle read-write concurrency, flash memory particle media specification, RAID configuration, transmission rate, etc. .

Exemplarily, the node includes a flash granule, and the SSD 210 may determine the performance specification of the flash granule based on at least one of the following: the read/write concurrency of the flash granule, WAF, the medium specification of the flash granule, the RAID configuration, the transmission rate, and the like.

Exemplarily, the node includes a controller chip, or the node may also include other nodes in the SSD 210, which will not be listed here.

It can be seen that the SSD performance specification is determined based on the performance of the node with the smallest performance on the data transmission path. This determination method takes into account the structure of the SSD, so that the obtained SSD performance is more accurate.

In some embodiments, SSD 210 may determine performance specifications based on operating parameters using a trained performance model. Specifically, some or all of the operating parameters may be input into the trained performance model, so that the output of the trained performance model is the performance specification.

The trained performance model may be obtained through training using a training data set, wherein the training data set may include multiple data items, and each data item may include operating parameters and corresponding performance specifications. Optionally, the operating parameters and actual performance specifications obtained at 201 may also be added to the training data set to update the performance model.

Embodiments of the present disclosure do not limit the network structure of the performance model, for example, it may be a neural network model including multiple convolutional layers, for example, it may include a regression tree (regression tree) or a decision tree structure.

Alternatively, a first performance model may be used to determine a steady-state performance specification, and a second performance model may be used to determine a maximum performance specification. Optionally, the third performance model may be used to determine IOPS, and the fourth performance model may be used to determine bandwidth. Embodiments of the present disclosure do not limit this.

It can be seen that using a pre-trained performance model to determine the performance of SSD disks can make full use of existing data and improve the efficiency of determining performance specifications.

It should be noted that although IOPS and bandwidth are used as performance specifications above, it can be understood that the performance specifications listed above are only illustrative. In actual applications, performance specifications may also include response time, queue depth, and queue per second. Other parameters that characterize SSD performance, such as the number of full times, are not limited in this disclosure.

In process 200, SSD 210 sends 203 performance specifications to host 220.

In some examples, the performance specification may be sent through a second specific field. For example a second specific field may be predefined for performance specification. In some examples, the second flag field may be used to indicate that the next transmission is a performance specification, and the performance specification is sent immediately after the second flag field. In this way, the embodiment of the present disclosure can ensure that the performance specification sent by the SSD 210 is correctly recognized by the host 220 by customizing the interface field.

In process 200, host 220 allocates 204 read and write operations based on performance specifications.

Based on the descriptions in 201 to 203 above, the host 220 may receive performance specifications from at least one storage device 110 (such as SSD 210) in the storage system 101. In some examples, the host 220 can receive respective performance specifications from multiple storage devices 110 (such as SSD 210), and then the host 220 can distribute read and write operations based on the respective performance specifications of multiple storage devices 110 (such as SSD 210) . This can optimize the data layout and improve the overall performance.

In this way, the host 220 can fully consider the disk performance of each SSD, realize the effective utilization of SSD disks, and ensure the overall performance of reading and writing.

Through the process 200 described in conjunction with FIG. 2 , the SSD in the embodiment of the present disclosure can determine the performance specification based on the read and write operations in the actual scene instead of the test environment, so that the obtained performance specification is more accurate. And in the determination process, the influence of the internal controller algorithm on the performance of the disk is also considered, so as to fully tap the ability of the disk and make the performance of the disk fully utilized.

It should be understood that in the embodiments of the present disclosure, "first", "second", "third" and so on are only used to indicate that multiple objects may be different, but it does not exclude that two objects are the same. "First", "second", "third" and so on should not be construed as any limitations on the embodiments of the present disclosure.

It should also be understood that the divisions of methods, situations, categories and embodiments in the embodiments of the present disclosure are only for the convenience of description, and should not constitute special limitations. Logical cases can be combined with each other.

It should also be understood that the above content is only for helping those skilled in the art to better understand the embodiments of the present disclosure, rather than limiting the scope of the embodiments of the present disclosure. Those skilled in the art may make various modifications or changes or combinations based on the above contents. Such modifications, changes or combined solutions are also within the scope of the embodiments of the present disclosure.

It should also be understood that the above description focuses on the differences between the various embodiments, and the same or similar points may refer to or learn from each other, and for the sake of brevity, details are not repeated here.

FIG. 3 shows a schematic block diagram of a data processing device 300 according to an embodiment of the present disclosure. The apparatus 300 may be implemented as the storage device 110 or a part of the storage device 110 as shown in FIG. 1 .

The apparatus 300 may include an acquiring module 310 , a receiving module 320 , a determining module 330 and a sending module 340 . The obtaining module 310 is configured to obtain operating parameters of multiple read and write operations performed within a period of time. The determining module 330 is configured to determine performance specifications of the storage device based on the operating parameters. The sending module 340 is configured to send the performance specification to a task distribution device outside the storage system.

In some embodiments, the determining module 330 may be configured to determine the performance specification based on the operating parameters in response to receiving a request for the performance of the storage device from the task allocation device. It can be understood that the receiving module 320 shown in FIG. 3 may be configured to receive the request for the performance of the storage device from the task allocation device.

In some embodiments, the determining module 330 may be configured to: if the number of multiple read and write operations exceeds the number threshold, or if the time between the current time point and the last time the storage device sends the performance specification to the task allocation device The length exceeds a time threshold, and a performance specification is determined based on the operating parameters.

Optionally, the determination module 330 may be configured to determine the node performance specification of each node in the storage device based on the operating parameters, and the node includes at least one of an interface, a channel, a flash memory particle, and a controller chip; and based on the node performance specification of each node Specifications, to determine performance specifications.

Optionally, the determination module 330 may be configured to input operating parameters into the trained performance model to obtain performance specifications.

Exemplarily, the running parameters include at least one of the following: read and write characteristic parameters, disk state parameters, or algorithm state parameters. Optionally, the read and write characteristic parameters include at least one of the following: read and write times, read and write size, read and write heat, or read and write randomness; the parameters of the disk state include parameters about at least one of the following: read and write Concurrency, write amplification factor, space utilization, controller gear, or media specifications of flash memory particles; the parameters of the algorithm state include indication information of whether the garbage collection GC algorithm is enabled, and/or RAID configuration.

Exemplarily, the performance specification includes a steady state performance specification and/or a maximum performance specification. The steady-state performance specification includes at least one of the following: steady-state read IOPS, steady-state read bandwidth, steady-state write IOPS, or steady-state write bandwidth. The maximum performance specification includes at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.

The device 300 in FIG. 3 can be used to implement the processes described above in conjunction with the SSD 210 in FIG. 2 , and for the sake of brevity, details are not repeated here.

Referring to FIG. 1 , each module in the device 300 can be implemented in the interface logic 112 and the controller 114 . Fig. 4 shows a schematic diagram of a module distribution 400 in which modules in the apparatus 300 are implemented in the storage device 110 according to some embodiments of the present disclosure. As shown in FIG. 4 , the receiving module 320 and the sending module 340 are implemented in the interface logic 112 , and the obtaining module 310 and the determining module 330 are implemented in the controller 114 . In addition, it can be understood that the storage device 110 in FIG. 4 may also include multiple flash memory particles and corresponding multiple channels, which are not shown in FIG. 4 for simplicity.

It can be understood that the names of the various modules shown in FIG. 3 or FIG. 4 are only for illustration. For example, the acquisition module 310 may also be called a statistics module, and the determination module 330 may also be called a performance evaluation module. For example, the receiving module 320 may also be called a host request performance interface module, and the sending module 340 may also be called a performance specification feedback interface module. Embodiments of the present disclosure are not limited to this.

Fig. 5 shows a schematic block diagram of an information device 500 according to an embodiment of the present disclosure. The apparatus 500 may be implemented as the task allocation device 102 as shown in FIG. 1 or a part of the task allocation device 102 .

The apparatus 500 may include a sending module 510 , a receiving module 520 and an allocating module 530 . The receiving module 520 is configured to receive performance specifications from at least one storage device in the storage system. The allocation module 530 is configured to allocate read and write operations to the at least one storage device based at least on the performance specification of the at least one storage device.

The sending module 510 of the apparatus 500 may be configured to send a request for the performance of the at least one storage device to the at least one storage device.

The device 500 in FIG. 5 can be used to implement the processes described above in conjunction with the host 220 in FIG. 2 , and for the sake of brevity, details will not be described here.

In addition, it can be understood that the division of modules or units in the embodiments of the present disclosure is schematic, and is only a logical function division, and there may be other division methods in actual implementation. In addition, in the disclosed embodiments Each functional unit can be integrated into one unit, or physically exist separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Fig. 6 shows a schematic block diagram of an example device 600 that may be used to implement embodiments of the present disclosure. Device 600 may be implemented to include SSD 210 and/or host 220 shown in FIG. 1 .

As shown in the figure, the device 600 includes a central processing unit (Central Processing Unit, CPU) 601, a read-only memory (Read-Only Memory, ROM) 602, and a random access memory (Random Access Memory, RAM) 603. The CPU 601 can perform various appropriate actions and processes according to computer program instructions stored in the RAM 602 and/or RAM 603 or loaded from the storage unit 608 into the ROM 602 and/or RAM 603. In the ROM 602 and/or the RAM 603, various programs and data necessary for the operation of the device 600 can also be stored. The CPU 601 and the ROM 602 and/or RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to the bus 604 .

Multiple components in the device 600 are connected to the I/O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc. ; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 609 allows the device 600 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks. Illustratively, SSD 210 as shown in FIG. 1 may be implemented as part of storage unit 608.

CPU 601 may be various general and/or special purpose processing components having processing and computing capabilities. Some examples that can be implemented as include, but are not limited to, Graphics Processing Unit (Graphics Processing Unit, GPU), various dedicated artificial intelligence (Artificial Intelligence, AI) computing chips, various computing units that run machine learning model algorithms, digital signal A processor (Digital Signal Processor, DSP), and any suitable processor, controller, microcontroller, etc., may accordingly be referred to as a computing unit.

Exemplarily, the device 600 in FIG. 6 can be implemented as various electronic devices, such as mobile handsets, desktop computers, laptop computers, notebook computers, tablet computers, personal communication system devices, personal navigation devices, personal digital assistants ( Personal Digital Assistant, PDA), e-book equipment, game equipment, Internet of Things (Internet of Things, IoT) equipment, vehicle equipment, aircraft, virtual reality (Virtual Reality, VR) equipment, augmented reality (Augmented Reality, AR) equipment, Wearable devices, TVs, cloud servers, data centers, etc. Embodiments of the present disclosure are not limited to this.

Embodiments of the present disclosure also provide a chip or a system chip, and the chip may include an input interface, an output interface, and a processing circuit. In the embodiments of the present disclosure, the interaction of signaling or data may be completed by the input interface and the output interface, and the generation and processing of signaling or data information may be completed by the processing circuit.

Embodiments of the present disclosure also provide a computer program product containing instructions, which, when run on a computer, cause the computer to execute the methods and functions involved in any of the above embodiments.

Embodiments of the present disclosure also provide a computer-readable storage medium on which computer instructions are stored, and when a processor executes the instructions, the processor is made to execute the methods and functions involved in any of the above embodiments.

In general, the various embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor or other computing device. While various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flowcharts, or using some other pictorial representation, it should be understood that the blocks, devices, systems, techniques or methods described herein can be implemented as, without limitation, Exemplary, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof.

The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product comprises computer-executable instructions, eg included in program modules, which are executed in a device on a real or virtual processor of a target to perform the process/method as above with reference to the accompanying drawings. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In various embodiments, the functionality of the program modules may be combined or divided as desired among the program modules. Machine-executable instructions for program modules may be executed within local or distributed devices. In a distributed device, program modules may be located in both local and remote storage media.

Computer program codes for implementing the methods of the present disclosure may be written in one or more programming languages. These computer program codes can be provided to processors of general-purpose computers, special-purpose computers, or other programmable data processing devices, so that when the program codes are executed by the computer or other programmable data processing devices, The functions/operations specified in are implemented. The program code may execute entirely on the computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server.

In the context of the present disclosure, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagated signals, such as carrier waves, infrared signals, and the like.

A computer readable medium may be any tangible medium that contains or stores a program for or related to an instruction execution system, apparatus, or device. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More detailed examples of computer-readable storage media include electrical connections with one or more wires, portable computer diskettes, hard disks, random storage access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash), optical storage, magnetic storage, or any suitable combination thereof.

In addition, while operations of methods of the present disclosure are depicted in a particular order in the figures, this does not require or imply that operations must be performed in that particular order, or that all illustrated operations must be performed, to achieve desirable results. Conversely, the steps depicted in the flowcharts may be performed in an altered order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution. It should also be noted that the features and functions of two or more devices according to the present disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided to be embodied by a plurality of devices.

Having described various implementations of the present disclosure, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The choice of terminology used herein is intended to well explain the principles of each implementation, practical applications or improvements to technologies in the market, or to enable other ordinary skilled persons in the technical field to understand the various implementations disclosed herein.

Claims

A data processing method, comprising:

The storage device acquires operating parameters of multiple read and write operations performed within a period of time;

the storage device determines performance specifications of the storage device based on the operating parameters; and

The storage device sends the performance specification to a task distribution device external to the storage system.
The method of claim 1, wherein the storage device determining the performance specification based on the operating parameters comprises:

The storage device determines the performance specification based on the operating parameters in response to receiving a request from the task assignment device for performance of the storage device.
The method of claim 1, wherein the storage device determining the performance specification based on the operating parameters comprises:

If the number of times of the multiple read and write operations exceeds the number threshold, or if the time length between the current time point and the last time point when the storage device sends the performance specification to the task allocation device exceeds the time threshold, the The storage device determines the performance specification based on the operating parameters.
The method according to any one of claims 1 to 3, wherein said storage device determining said performance specification based on said operating parameters comprises:

The storage device determines node performance specifications of each node in the storage device based on the operating parameters, and the nodes include at least one of an interface, a channel, a flash memory particle, or a controller chip; and

The storage device determines the performance specification based on the node performance specification of each node.
The method according to any one of claims 1 to 3, wherein said storage device determining said performance specification based on said operating parameters comprises:

The storage device inputs the operating parameters to a trained performance model to obtain the performance specification.
The method according to any one of claims 1 to 5, wherein the operating parameters include at least one of the following: read and write characteristic parameters, disk state parameters, or algorithm state parameters.
The method according to claim 6, wherein the read-write characteristic parameters include at least one of the following: read-write times, read-write size, read-write heat, or read-write randomness; the parameters of the disk state include the following At least one parameter: read and write concurrency, write amplification factor, space utilization, controller gear, or media specification of flash memory particles; the parameters of the algorithm state include at least one of the following: whether the garbage collection GC algorithm is enabled , or a redundant array of independent disks RAID configuration.
The method of any one of claims 1 to 7, wherein the performance specification includes at least one of a steady state performance specification and a maximum performance specification.
The method according to claim 8, wherein the steady-state performance specification includes at least one of the following: steady-state read input and output times per second IOPS, steady-state read bandwidth, steady-state write IOPS, or steady-state write bandwidth;

The maximum performance specification includes at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.
A data processing method, comprising:

The task allocation device receives performance specifications from at least one storage device in the storage system; and

The task allocation device allocates read and write operations to the at least one storage device based at least on the performance specification of the at least one storage device.
The method of claim 10, further comprising:

The task allocating device sends a request for the performance of the at least one storage device to the at least one storage device.
The method of claim 10 or 11, wherein the performance specification includes at least one of a steady state performance specification and a maximum performance specification.
The method according to claim 12, wherein the steady-state performance specification includes at least one of the following: steady-state read IOPS, steady-state read bandwidth, steady-state write IOPS, or steady-state write bandwidth, and wherein The maximum performance specification includes at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.
A data processing device comprising

An acquisition module configured to acquire operating parameters of multiple read and write operations performed within a period of time;

a determining module configured to determine performance specifications of the storage device based on the operating parameters; and

A sending module configured to send the performance specification to a task allocation device outside the storage system.
The apparatus of claim 14, wherein the determining module is configured to:

The performance specification is determined based on the operating parameters in response to receiving a request from the tasking device for performance of the storage device.
The apparatus of claim 14, wherein the determining module is configured to:

If the number of times of the multiple read and write operations exceeds the number threshold, or if the time length between the current time point and the last time point when the storage device sends the performance specification to the task allocation device exceeds the time threshold, based on the The operating parameters described above determine the performance specifications.
The apparatus according to any one of claims 14 to 16, wherein the determining module is configured to:

determining a node performance specification of each node in the storage device based on the operating parameters, the node including at least one of an interface, a channel, a flash memory particle, or a controller chip; and

The performance specification is determined based on the node performance specification of the respective nodes.
The apparatus according to any one of claims 14 to 16, wherein the determining module is configured to:

The operating parameters are input to the trained performance model to obtain the performance specification.
The device according to any one of claims 14 to 18, wherein the operating parameters include at least one of the following: read and write characteristic parameters, disk state parameters, or algorithm state parameters.
The device according to claim 19, wherein the read-write characteristic parameters include at least one of the following: read-write times, read-write size, read-write heat, or read-write randomness; the parameters of the disk state include the following At least one parameter: read and write concurrency, write amplification factor, space utilization, controller gear, or media specification of flash memory particles; the parameters of the algorithm state include at least one of the following: whether the garbage collection GC algorithm is enabled , or a redundant array of independent disks RAID configuration.
The apparatus of any one of claims 14 to 20, wherein the performance specification includes at least one of a steady state performance specification and a maximum performance specification.
The device according to claim 21, wherein the steady-state performance specification includes at least one of the following: steady-state read IOPS, steady-state read bandwidth, steady-state write IOPS, or steady-state write bandwidth;

The maximum performance specification includes at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.
A data processing device, comprising:

a receiving module configured to receive performance specifications from at least one storage device in the storage system; and

An allocation module configured to allocate read and write operations to the at least one storage device based at least on the performance specification of the at least one storage device.
The apparatus according to claim 23, further comprising a sending module configured to send a request for the performance of the at least one storage device to the at least one storage device.
The apparatus of claim 23 or 24, wherein the performance specification comprises at least one of a steady state performance specification and a maximum performance specification.
The apparatus according to claim 25, wherein the steady-state performance specification includes at least one of the following: steady-state read IOPS, steady-state read bandwidth, steady-state write IOPS, or steady-state write bandwidth, and wherein The maximum performance specification includes at least one of the following: maximum read IOPS, maximum read bandwidth, maximum write IOPS, or maximum write bandwidth.
A data processing device comprising a processor and a memory, the memory having computer instructions stored thereon, which, when executed by the processor, cause the device to perform any one of claims 1 to 9, or A method as claimed in any one of claims 10 to 13.
A computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the computer-readable instructions according to any one of claims 1 to 9, or claims 10 to 9 are implemented. The method described in any one of 13.
A computer program product having computer-executable instructions embodied thereon which, when executed, implements any one of claims 1 to 9, or any one of claims 10 to 13 the method described.