WO2024077993A1

WO2024077993A1 - Resource allocation method and apparatus, storage medium, and program product

Info

Publication number: WO2024077993A1
Application number: PCT/CN2023/100421
Authority: WO
Inventors: 张嘉伟; 张; 陈俊伟
Original assignee: 华为技术有限公司
Priority date: 2022-10-14
Filing date: 2023-06-15
Publication date: 2024-04-18
Also published as: CN117931416A

Abstract

The present application relates to a resource allocation method and apparatus, a storage medium, and a program product. The method comprises: acquiring the size of an available reserved space of a first object and the size of an available reserved space of a second object; and according to a characteristic parameter of a user load and a type parameter of a garbage collection algorithm, adjusting the size of an actual reserved space of the first object and the size of an actual reserved space of the second object. According to the resource allocation method of embodiments of the present application, when a system comprises a first object and a second object, a configuration method for dynamically adjusting the reserved spaces of two objects when the system is running reduces the influence of write amplification on the system performance. Moreover, there is no need to specially design an interface, such that there is no need to re-design a software stack for the interface, thereby reducing data processing costs and operation and maintenance costs, and optimizing the user experience.

Description

Resource allocation method, device, storage medium and program product

This application claims priority to the Chinese patent application filed with the China Patent Office on October 14, 2022, with application number 202211262519.8 and application name “Resource allocation method, device, storage medium and program product”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of data storage, and in particular to a resource allocation method, device, storage medium and program product.

Background technique

The two-layer log (log-on-log) structure is currently a mainstream storage system architecture. The storage system of this architecture includes two objects based on the log structure, and each object supports the garbage collection (GC) algorithm to ensure that the storage system can implement a sequential write mechanism. The more common objects are the system layer of software and the device layer of hardware (including solid state disks (SSD)). In order to reduce the adverse effects of write amplification (WA) on the performance of the storage system, a certain amount of over-provisioning (OP) is generally set in each of the two objects to serve garbage collection and ensure the performance of the storage system. The larger the reserved space, the more friendly it is to write amplification and the higher the performance of the storage system.

The reserved space size of the device layer and system layer of some storage systems is determined after the user purchases them. It will not change during the subsequent operation of the storage system and is not visible to the user. There may be a situation where the reserved space at the system layer is limited, resulting in heavy write pressure, which in turn leads to poor network connection performance from the system layer to the device layer of the storage system. Other storage systems have improved the hardware devices at the device layer, and actively exposed the improved interface to the system layer through the device layer, so that there is almost no reserved space at the device layer. Instead, the system layer controls the reserved space allocation of the system layer and the device layer, which in turn increases the size of the reserved space at the system layer. However, because it leaves the management of hardware to software decisions, it is necessary to redesign the software stack for hardware devices, resulting in high costs for the entire software stack ecosystem and subsequent operation and maintenance.

Summary of the invention

In view of this, a resource allocation method, device, storage medium and program product are proposed. According to the resource allocation method of the embodiment of the present application, when the system includes a first object and a second object, the configuration mode of the reserved space of the second object can also be dynamically adjusted when the system is running, so that the storage resource allocation mode is more flexible and the impact of write amplification on system performance is reduced. And there is no need to specially design the interface, so there is no need to redesign the software stack for the interface, which can reduce data processing costs and operation and maintenance costs, and optimize user experience.

In a first aspect, an embodiment of the present application provides a resource allocation method, the method comprising: obtaining the size of the available reserved space of a first object and the size of the available reserved space of a second object; adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to characteristic parameters of the user load and type parameters of the garbage collection algorithm.

According to the resource allocation method of the embodiment of the present application, by obtaining the size of the available reserved space of the first object and the size of the available reserved space of the second object, the size of the available reserved space determined by the hardware parameters and actual usage of the first object and the second object can be determined; the size of the actual reserved space of the first object and the size of the actual reserved space of the second object are adjusted according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm, so that the actual reserved space size of the first object is not limited by the hardware parameters of the first object, and the actual reserved space size of the second object is not limited by the hardware parameters of the first object, and the flexible adjustment of the actual reserved space size is realized at the software level; when the system includes the first object and the second object, the configuration method of the actual reserved space of the second object can also be dynamically adjusted when the system is running, so that the resource allocation method is more flexible; by integrating the characteristic parameters of the user load and the type parameters of the garbage collection algorithm and other factors that affect the system, reasonable allocation of resources is realized, so the impact of write amplification on system performance can be reduced. And the allocation does not require special design of the interface, so there is no need to redesign the corresponding software stack, which can reduce data processing costs and operation and maintenance costs, and optimize user experience.

According to the first aspect, in a first possible implementation of the resource allocation method, the first object and the second object are set in a storage system, the storage system runs the garbage collection algorithm, the first object includes the system layer of the storage system, the second object includes the device layer of the storage system, the available reserved space of the first object includes the hot backup space, the initial reserved space and the unused space in the user-visible space of the system layer; the available reserved space of the second object includes the initial reserved space of the device layer; wherein the hot backup space, the initial reserved space of the system layer and the initial reserved space of the device layer remain unchanged.

In this way, the resource allocation method of the embodiment of the present application can be applied to a storage system based on a general hardware device. Since there is no need to customize the hardware device, there is no need to design and maintain the corresponding software stack, which can reduce the dependence of the resource allocation method on the hardware device and improve the universality of the resource allocation method. Under the premise of specific system-level software configuration and device-level hardware configuration, the network connection between different objects in the storage system can maintain good performance, improve the utilization of the device-level space, reduce write amplification, extend the life of the device-level hardware, and reduce costs.

According to the first aspect, in a second possible implementation of the resource allocation method, the first object and the second object are set in a storage system, the storage system runs the garbage collection algorithm, the first object includes the system layer of the storage system, the second object includes the application layer of the storage system, the available reserved space of the first object includes the hot backup space, the initial reserved space and the unused space in the user-visible space of the system layer; the available reserved space of the second object includes the initial reserved space of the application layer; wherein the hot backup space and the initial reserved space of the system layer remain unchanged.

In this way, the resource allocation method of the embodiment of the present application can be applied to any storage system with a double-layer log structure, which can further improve the universality of the resource allocation method and expand the application scenarios of the resource allocation method.

According to the first aspect, or any possible implementation of the first aspect above, in a third possible implementation of the resource allocation method, before adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm, the method also includes: determining the user input/output access objects and frequency within a preset time window; sorting the access objects according to the frequency and constructing a statistical histogram; based on the statistical histogram, determining the linear relationship between the frequency and the sorting sequence number of the access objects; determining the characteristic parameters of the user load according to the slope of the linear relationship.

In this way, the characteristic parameters of the user load can be determined in real time. The characteristic parameters of the user load can be determined at any time before or during the operation of the system, which will not affect the normal operation of the storage system, and is convenient for users to determine the user load situation. It also provides the possibility of adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object using the characteristic parameters of the user load.

According to the first aspect, or any possible implementation of the first aspect above, in a fourth possible implementation of the resource allocation method, adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm includes: selecting a configuration method that matches the sum of the spaces of the size of the available reserved space of the first object and the size of the available reserved space of the second object, the characteristic parameters and the type parameters from a plurality of pre-set configuration methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object as the optimal configuration method; and adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration method.

In this way, the best configuration method can be quickly determined during system operation, thereby improving the efficiency from obtaining the available reserved space size of the first object and the available reserved space size of the second object to adjusting the actual reserved space size of the first object and the actual reserved space size of the second object.

According to the fourth possible implementation manner of the first aspect, in the fifth possible implementation manner of the resource allocation method, the method also includes: determining a value range of the total space according to the hot backup space, the initial reserved space, the user visible space, and the available reserved space of the second object; for each numerical value in the value range of the total space, determining multiple division methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object under the numerical value; determining multiple configuration methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to multiple characteristic parameters of the user load, multiple types of parameters of the garbage collection algorithm, the total space, and the multiple division methods.

In this way, multiple configuration modes of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object can be determined. Since characteristic parameters of user load related to storage system performance, type parameters of garbage collection algorithm, total space and multiple division modes are used as the basis for determining multiple configuration modes, the storage system performance is guaranteed under the determined configuration mode.

According to a fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the resource allocation method, the multiple configuration manners of determining the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the multiple characteristic parameters of the user load, the multiple type parameters of the garbage collection algorithm, the space sum, and the multiple division methods include: determining multiple combinations of the characteristic parameters, the type parameters, and the space sum according to the multiple characteristic parameters of the user load, the multiple type parameters of the garbage collection algorithm, and the space sum, wherein at least one of the characteristic parameters, the type parameters, and the space sum in different combinations is different; for each combination of the characteristic parameters, the type parameters, and the space sum, respectively determining the write amplification factor of each division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object; determining the division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object corresponding to the smallest write amplification factor in each combination as a configuration manner of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.

In this way, multiple configuration modes of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object can be determined. The multiple configuration modes finally determined are all optimal in performance, thus saving the cost of storing multiple configuration modes.

According to any one of the fourth to sixth possible implementations of the first aspect, in the seventh possible implementation of the resource allocation method, under the optimal configuration mode, when the size of the actual reserved space of the first object decreases and the size of the actual reserved space of the second object increases, adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode includes at least one of the following: lowering a watermark parameter of the garbage collection algorithm, and the garbage collection algorithm is started when the value of the watermark parameter is greater than the size of the actual reserved space of the first object; increasing a speed parameter of the garbage collection algorithm, and the speed parameter indicates the garbage collection speed when the storage system executes the garbage collection algorithm; and reducing the startup delay of the garbage collection algorithm.

In this way, when the optimal configuration indicates that the size of the actual reserved space of the first object is reduced and the size of the actual reserved space of the second object is increased, the storage system can complete the adjustment of the actual reserved space of the first object and the second object as soon as possible, reducing the time required for the storage system to adapt to the new configuration mode and improving the performance of the storage system. By adopting multiple ways to complete the adjustment, the adjustment method is more flexible.

According to any one of the fourth to seventh possible implementations of the first aspect, in an eighth possible implementation of the resource allocation method, under the optimal configuration mode, when the size of the actual reserved space of the first object increases and the size of the actual reserved space of the second object decreases, adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode includes at least one of the following: increasing a waterline parameter of the garbage collection algorithm, and the garbage collection algorithm is started when the value of the waterline parameter is greater than the size of the actual reserved space of the first object; reducing a speed parameter of the garbage collection algorithm, and the speed parameter indicates the garbage collection speed when the storage system executes the garbage collection algorithm; and increasing the startup delay of the garbage collection algorithm.

In this way, when the optimal configuration indicates that the size of the actual reserved space of the first object is increased and the size of the actual reserved space of the second object is decreased, the storage system can complete the adjustment of the actual reserved space of the first object and the second object as soon as possible, reducing the time required for the storage system to adapt to the new configuration mode and improving the performance of the storage system. By adopting multiple ways to complete the adjustment, the adjustment method is more flexible.

In a second aspect, an embodiment of the present application provides a resource allocation device, comprising: an acquisition module for acquiring the size of an available reserved space of a first object and the size of an available reserved space of a second object; an adjustment module for adjusting the size of an actual reserved space of the first object and the size of an actual reserved space of the second object according to characteristic parameters of a user load and type parameters of a garbage collection algorithm.

According to the second aspect, in a first possible implementation of the resource allocation device, the first object and the second object are set in a storage system, the storage system runs the garbage collection algorithm, the first object includes the system layer of the storage system, the second object includes the device layer of the storage system, the available reserved space of the first object includes the hot backup space, the initial reserved space and the unused space in the user-visible space of the system layer; the available reserved space of the second object includes the initial reserved space of the device layer; wherein the hot backup space, the initial reserved space of the system layer and the initial reserved space of the device layer remain unchanged.

According to the second aspect, in a second possible implementation of the resource allocation device, the first object and the second object are set in a storage system, the storage system runs the garbage collection algorithm, the first object includes the system layer of the storage system, the second object includes the application layer of the storage system, the available reserved space of the first object includes the hot backup space, the initial reserved space and unused space in the user-visible space of the system layer; the available reserved space of the second object includes the initial reserved space of the application layer; wherein the hot backup space and the initial reserved space of the system layer remain unchanged.

According to the second aspect, or any possible implementation of the second aspect above, in a third possible implementation of the resource allocation device, the device also includes: a first determination module, used to determine the user input/output access objects and frequency within a preset time window; a construction module, used to sort the access objects according to the frequency and construct a statistical histogram; a second determination module, used to determine the linear relationship between the frequency and the sorting sequence number of the access object based on the statistical histogram; a third determination module, used to determine the characteristic parameters of the user load according to the slope of the linear relationship.

According to the second aspect, or any possible implementation of the second aspect above, in a fourth possible implementation of the resource allocation device, the adjustment module includes: a selection unit, used to select a configuration method that matches the sum of the space of the available reserved space of the first object and the size of the available reserved space of the second object, the characteristic parameters, and the type parameters from a plurality of pre-set configuration methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object as the optimal configuration method; an adjustment unit, used to adjust the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration method.

According to a fourth possible implementation manner of the second aspect, in a fifth possible implementation manner of the resource allocation device, the device also includes: a fourth determination module, used to determine the value range of the total space according to the hot backup space, the initial reserved space, the user visible space, and the available reserved space of the second object; a fifth determination module, used to determine, for each value in the value range of the total space, multiple division methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object under the value; a sixth determination module, used to determine multiple configuration methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to multiple characteristic parameters of the user load, multiple types of parameters of the garbage collection algorithm, the total space, and the multiple division methods.

According to a fifth possible implementation manner of the second aspect, in a sixth possible implementation manner of the resource allocation device, the sixth determination module includes: a first determination unit, used to determine multiple combinations of the characteristic parameters, the type parameters, and the space sum according to multiple characteristic parameters of the user load, multiple type parameters of the garbage collection algorithm, and the space sum, wherein at least one of the characteristic parameters, the type parameters, and the space sum in different combinations is different; a second determination unit, used to determine the write amplification factor of each division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object for each combination of the characteristic parameters, the type parameters, and the space sum; a third determination unit, used to determine the division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object corresponding to the smallest write amplification factor in each combination as a configuration method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.

According to any one of the fourth to sixth possible implementations of the second aspect, in the seventh possible implementation of the resource allocation device, under the optimal configuration mode, when the size of the actual reserved space of the first object decreases and the size of the actual reserved space of the second object increases, adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode includes at least one of the following: lowering a watermark parameter of the garbage collection algorithm, and the garbage collection algorithm is started when the value of the watermark parameter is greater than the size of the actual reserved space of the first object; increasing a speed parameter of the garbage collection algorithm, and the speed parameter indicates the garbage collection speed when the storage system executes the garbage collection algorithm; and reducing the startup delay of the garbage collection algorithm.

According to any one of the fourth to seventh possible implementations of the second aspect, in an eighth possible implementation of the resource allocation device, under the optimal configuration mode, when the size of the actual reserved space of the first object increases and the size of the actual reserved space of the second object decreases, adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode includes at least one of the following: increasing a waterline parameter of the garbage collection algorithm, and the garbage collection algorithm is started when the value of the waterline parameter is greater than the size of the actual reserved space of the first object; reducing a speed parameter of the garbage collection algorithm, and the speed parameter indicates the garbage collection speed when the storage system executes the garbage collection algorithm; and increasing the startup delay of the garbage collection algorithm.

In a third aspect, an embodiment of the present application provides a resource allocation device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the resource allocation method of the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect when executing the instructions.

In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the resource allocation method of the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect.

In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in an electronic device, the processor in the electronic device executes the resource allocation method of the above-mentioned first aspect or one or several of the multiple possible implementation methods of the first aspect.

These and other aspects of the present application will become more apparent from the following description of the embodiment(s).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

FIG1 shows a schematic diagram of a typical two-layer log structure storage system.

FIG. 2 shows an example of a method for allocating reserved space at the system layer and the device layer of a two-layer log structure storage system.

FIG. 3 a shows an exemplary application scenario of the resource allocation method according to an embodiment of the present application.

FIG. 3 b shows an exemplary application scenario of the resource allocation method according to an embodiment of the present application.

FIG4 is a schematic diagram showing a process of a resource allocation method according to an embodiment of the present application.

FIG5 is a schematic diagram showing a method for obtaining characteristic parameters of a user load according to an embodiment of the present application.

FIG. 6 shows an example of a functional relationship between a write amplification factor of a single object and a reserved space size according to an embodiment of the present application.

FIG. 7 shows an example of a functional relationship between a write amplification factor of a storage system and reserved space sizes of two objects according to an embodiment of the present application.

FIG. 8 is a schematic diagram showing how to adjust the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to an embodiment of the present application.

FIG. 9 is a schematic diagram showing the structure of a resource allocation device according to an embodiment of the present application.

FIG. 10 shows an exemplary structural diagram of a resource allocation device according to an embodiment of the present application.

Detailed ways

Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

In addition, in order to better illustrate the present application, numerous specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

As shown in Figure 1, the storage system includes two log-structured objects, namely the system layer and the device layer. Each object supports the garbage collection algorithm to ensure that the storage system can implement a sequential write mechanism. This mechanism will cause write amplification, also known as write amplification, that is, the actual amount of physical data written is multiple times the theoretical value of the amount of data written. The reason is that, assuming that an 8KB data is to be written, but there is no free space at this time, but there is invalid data that can be erased, and the erasure is in blocks, so it is necessary to first write a block of data to the cache space or reserved space, then erase the block, and then write the 8KB new data. That is, originally 8KB of data was written, but it caused the entire block (512KB) to be written, that is, 64 times amplification. The erasure process is garbage collection. It can be seen that the larger the write amplification factor, the lower the data writing efficiency, which reduces the data storage efficiency of the storage system.

In order to reduce the adverse effects of write amplification on the performance of the storage system, a certain amount of over-provisioning (OP) is generally set in the two objects to serve garbage collection and ensure the performance of the storage system. The space at the system layer is mainly used to store the garbage collection algorithm, user data, metadata at the system layer, and the garbage collection algorithm for serving the system layer. The part used to serve the garbage collection algorithm at the system layer can be called the reserved space at the system layer. The space at the device layer is mainly used to store the garbage collection algorithm, host data, metadata at the device layer, and the space used to serve the garbage collection algorithm at the device layer. The part used to serve the garbage collection algorithm at the device layer can be called the reserved space at the device layer. The larger the reserved space, the more friendly it is to write amplification and the higher the system performance, but it also brings the problem of high cost.

FIG. 2 shows an example of a reserved space allocation method for the system layer and the device layer of a dual-layer log structure storage system. In the example of FIG. 2 , before the system is run, a fixed reserved space is set for the system layer and the device layer respectively according to user requirements. Among them, the space of the device layer may include the device visible space and the fixed reserved space of the device layer. The space of the system layer may include the system visible space, the disk array, the hot backup space and the fixed reserved space of the system layer. The size of the device visible space is the same as the size of the space of the system layer. The system visible space and the disk array are also visible to the user, so the sum of the system visible space and the disk array is also equal to the user visible space of the user layer.

In the example of Figure 2, for the device layer, the maximum value of its available reserved space is equal to the size of the fixed reserved space of the device layer. For the system layer, the maximum value of its available reserved space is equal to the sum of the hot backup space and the fixed reserved space of the system layer. The maximum values of the available reserved space of the device layer and the system layer will not change in the subsequent operation of the storage system and will not be visible to the user. In this case, there is a situation where the reserved space of the system layer is limited, resulting in a large write pressure, which in turn leads to poor network connection performance from the system layer to the device layer of the storage system.

Some other storage systems have improved the hardware devices of the device layer, and actively exposed the improved interface to the system layer through the device layer, so that there is almost no reserved space in the device layer. Instead, the reserved space allocation of the system layer and the device layer is entirely controlled by the system layer, which in disguise expands the size of the reserved space of the system layer. For example, a multi-stream solid state disk (MS-SSD) can provide multi-stream tags inside the SSD, bringing the data characteristics of the user tags into the SSD to achieve control from the user layer to the system layer to the device layer. Another example is an open channel solid state disk (OC-SSD), which exposes the parallel operation characteristics of the SSD to the user, and the user decides the address mapping, erasure, input/output scheduling, etc., to achieve control from the user layer to the system layer to the device layer. Another example is the zone namespaces solid state disk (ZNS-SSD), as a derivative type of open channel SSD, which proposes the concept of partitioning. The erase logic of the device layer partition is controlled by the system layer on the host side, so that the internal management of the SSD is controlled by the system layer, thereby eliminating the write amplification effect of garbage collection of the SSD and reducing the fixed reserved space on the SSD side.

The disadvantages of using the above-mentioned storage system with customized hardware devices are that, first, the customized hardware device itself provides a specially customized interface and exposes it to the system layer to achieve the goal of reducing the fixed reserved space, which is not applicable to general hardware devices. Therefore, the dependence on hardware is too strong and the interface is not universal. Second, the management of the underlying hardware is handed over to the upper layer for decision-making, which requires the redesign of the software stack of the system layer. The software stack has high ecological costs and subsequent maintenance costs, and the user's operation is complicated, which is extremely unfriendly to users.

In view of this, a resource allocation method, device, storage medium and program product are proposed. According to the resource allocation method of the embodiment of the present application, the configuration mode of the reserved space of the second object can also be dynamically adjusted when the system is running, so that the storage resource allocation mode is more flexible and the impact of write amplification on system performance is reduced. And there is no need to specially design the interface, so there is no need to redesign the software stack for the interface, which can reduce data processing costs and operation and maintenance costs and optimize user experience.

FIG. 3a and FIG. 3b respectively illustrate an exemplary application scenario of the resource allocation method according to an embodiment of the present application.

As shown in Figures 3a and 3b, the resource allocation method of an embodiment of the present application can be applied to a storage system with a double-layer log structure. The storage system may include two objects based on the log structure. Both objects are append-sequential write mechanisms and both require space to be freed up through garbage collection. Therefore, a garbage collection algorithm is provided.

In the example of FIG. 3a, the two objects may be a system layer and a device layer, respectively. The device layer may include a general solid-state hard disk, and the storage system may be a double-layer log structure storage system based on a solid-state hard disk. The system layer may include a system-visible space, a disk array (the sum of the system-visible space and the disk array is the user-visible space), a hot backup space, and an initial reserved space, wherein the hot backup space of the system layer may be, for example, the hot backup space of the system layer in FIG. 2, the initial reserved space may be the fixed reserved space of the system layer in FIG. 2, and the user-visible space may be the system-visible space and the disk array of the system layer in FIG. 2. One or more of the system-visible space, the disk array, and the hot backup space can be used to store garbage collection algorithms, user data, metadata, etc. The solid-state hard disk of the device layer may include a device-visible space and an initial reserved space, and the initial reserved space of the device layer may be, for example, the fixed reserved space of the device layer in FIG. 2. The device-visible space can be used to store garbage collection algorithms, user data, metadata, etc. The initial reserved space of the system layer, the hot backup space, and the unused portion of the user-visible space are collectively used as the available reserved space of the system layer. The initial reserved space of the device layer is used as the available reserved space of the device layer. The total space of the available reserved space of the system layer and the device layer (hereinafter referred to as the total space) is shared and allocated to the system layer and the device layer according to the requirements of the application scenarios during system operation to serve the garbage collection algorithms of the system layer and the device layer.

In the example of FIG3b, the two objects may be a system layer and an application layer, respectively. The application layer may include a key-value separated log structure application, and the storage system may be a two-layer log structure file system, such as a flash friendly file-system (F2FS). The system layer may include a system visible space, a disk array (the sum of the system visible space and the disk array is the user visible space), a hot backup space, and an initial reserved space, wherein the hot backup space of the system layer may be, for example, the hot backup space of the system layer in FIG2, the initial reserved space may be the fixed reserved space of the system layer in FIG2, and the user visible space may be the system visible space and the disk array of the system layer in FIG2. One or more of the system visible space, the disk array, and the hot backup space may be used to store garbage collection algorithms, user data, metadata, etc. The application layer may include an application visible space and an initial reserved space. The application visible space may be used to store garbage collection algorithms, application data, metadata, etc. The initial reserved space of the system layer, the hot backup space, and the unused portion of the user-visible space are collectively used as the available reserved space of the system layer. The initial reserved space of the application layer is used as the available reserved space of the application layer. The total space of the reserved space of the system layer and the application layer (hereinafter referred to as the total space) is shared and allocated to the system layer and the application layer according to the requirements of the application scenario during system operation to serve the garbage collection algorithms of the system layer and the application layer.

The storage system shown in Figures 3a and 3b can be set on a terminal device or a server. For example, the terminal device of the present application can be a smart phone, a netbook, a tablet computer, a laptop computer, a wearable electronic device (such as a smart bracelet, a smart watch, etc.), a TV, a virtual reality device, a speaker, electronic ink, etc.

As shown in FIG. 4 , in a possible implementation, the present application proposes a resource allocation method, the method comprising steps S41-S42:

Step S41, obtaining the size of the available reserved space of the first object and the size of the available reserved space of the second object.

Among them, the first object and the second object can be log-structured objects included in the system (such as the storage system described above). The first object may include a system layer, and the second object may include a device layer or an application layer. The available reserved space can be determined based on the hardware parameters and actual usage of the object, and the sum of the space allowed to be used for garbage collection on the object at the current moment. The size of the available reserved space of the first object and the size of the available reserved space of the second object can be added to obtain the total space. The total space can be shared by the first object and the second object, and is the resource to be allocated to the first object and the second object in an embodiment of the present application.

Step S42: adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm.

The characteristic parameters of the user load can indicate the user load situation, and the size of the reserved space required under different user loads may be different. The specific acquisition method can be found in the following text and the relevant descriptions of Figures 5-7. There can be many types of garbage collection algorithms, and the size of the reserved space required under different types of garbage collection algorithms may also be different. The type parameter of the garbage collection algorithm can be determined based on the system's configuration information for the garbage collection algorithm, and can be obtained through the interface provided by two objects. The acquisition method can be implemented based on the existing technology and will not be repeated here.

When the sum of the space, characteristic parameters, and type parameters of the size of the available reserved space of the first object and the size of the available reserved space of the second object are constant, the sum of the space is allocated to the space of the first object and the second object as the actual reserved space of the first object and the actual reserved space of the second object, respectively, wherein there can be multiple allocation methods, and the performance of the storage system may be different under each allocation method. In this way, the size of the actual reserved space of the first object and the size of the actual reserved space of the second object can be adjusted. The adjusted size of the available reserved space of the first object and the size of the available reserved space of the second object can be the most suitable for the current system, thereby improving the performance of the system.

The following describes how to determine the available reserved space of the second object when the second object is different.

In a possible implementation, the first object and the second object are set in a storage system, and the storage system runs the garbage collection algorithm.

The first object includes a system layer of the storage system, and the second object includes a device layer of the storage system.

The available reserved space of the first object includes the hot backup space of the system layer, the initial reserved space, and the unused space in the user visible space;

The available reserved space of the second object includes the initial reserved space of the device layer;

Among them, the hot backup space and the initial reserved space of the system layer and the initial reserved space of the device layer remain unchanged.

For example, when the first object includes the system layer of the storage system and the second object includes the device layer of the storage system, an example of the storage system can be shown in Figure 3a. The device layer can include general hardware devices, such as general solid-state drives, without using customized hardware devices, such as the multi-stream solid-state drive MS-SSD, open channel solid-state drive OC-SSD, partition namespace solid-state drive ZNS-SSD, etc. mentioned above.

At this time, the available reserved space of the first object may include the hot backup space of the system layer, the initial reserved space, and the unused space in the user-visible space. The hot backup space and the initial reserved space of the system layer may be set before the system is run, and therefore may be fixed. In this case, for the available reserved space of the first object, the variable part lies in the size of the unused space in the user-visible space of the system layer. Among them, the hot backup space of the system layer may be, for example, the hot backup space of the system layer in Figure 2, the initial reserved space may be the fixed reserved space of the system layer in Figure 2, and the user-visible space may be the system-visible space and disk array of the system layer in Figure 2.

The available reserved space of the second object may include the initial reserved space of the device layer. The initial reserved space of the device layer may be set before the system is run, and thus may be fixed. In this case, the available reserved space of the second object may also be fixed. The initial reserved space of the device layer may be, for example, the fixed reserved space of the device layer in FIG. 2.

Therefore, during the operation of the system, each time step S41 is executed, the size of the available reserved space of the first object needs to be obtained in real time, while the size of the available reserved space of the second object can be obtained once when step S41 is executed for the first time or before the system runs. When the size of the available reserved space of the second object needs to be obtained subsequently, the size of the available reserved space of the second object that has been obtained can be directly used.

The first object includes a system layer of the storage system, and the second object includes an application layer of the storage system.

The available reserved space of the second object includes the initial reserved space of the application layer;

Among them, the hot backup space and the initial reserved space of the system layer remain unchanged.

For example, when the first object includes the system layer of the storage system and the second object includes the application layer of the storage system, an example of the storage system can be seen in Figure 3b. At this time, the available reserved space of the first object can include the hot backup space of the system layer (such as the hot backup space of the system layer in Figure 2), the initial reserved space (such as the fixed reserved space of the system layer in Figure 2), and the unused space in the user-visible space (such as the system-visible space and disk array of the system layer in Figure 2). The hot backup space and the initial reserved space of the system layer can be set before the system is run, and therefore can be fixed. In this case, for the available reserved space of the first object, the variable part lies in the size of the unused space in the user-visible space of the system layer.

The available reserved space of the second object may include the initial reserved space of the application layer. The initial reserved space of the application layer may be set before the system is run and remain unchanged during the system operation. In this case, the available reserved space of the second object may also be fixed. Alternatively, the initial reserved space of the application layer may also be set to be variable according to user needs during the system operation. In this case, for the available reserved space of the second object, the variable part is the real-time size of the initial reserved space of the application layer. The initial reserved space of the application layer can be seen in the example in Figure 3b.

Therefore, during the operation of the system, each time step S41 is executed, the size of the available reserved space of the first object needs to be obtained in real time, and the size of the available reserved space of the second object can be obtained according to the system settings. If the initial reserved space of the application layer is fixed, the size of the available reserved space of the second object only needs to be obtained once when step S41 is executed for the first time or before the system runs. When the size of the available reserved space of the second object needs to be obtained subsequently, the size of the available reserved space of the second object that has been obtained can be directly used; if the size of the available reserved space of the second object is variable, the size of the available reserved space of the second object also needs to be obtained in real time.

In actual applications, the hardware storage device corresponding to the available reserved space may have some bad blocks, and it can be considered that the space corresponding to the bad blocks is occupied and cannot be erased. In this case, the available reserved space of the first object and the second object may change. Since bad blocks may appear at any time during the operation of the system, in order to make the determined optimal configuration method more accurate, the size of the available reserved space of the first object and the size of the available reserved space of the second object may be obtained once each time step S41 is executed. The present application does not limit the frequency of obtaining the size of the available reserved space of the first object and the size of the available reserved space of the second object.

The following is an exemplary method for obtaining characteristic parameters of user load according to an embodiment of the present application. Fig. 5 is a schematic diagram of a method for obtaining characteristic parameters of user load according to an embodiment of the present application.

As shown in FIG. 5 , in a possible implementation manner, before step S42, the method further includes:

Determine the user input/output access objects and frequencies within a preset time window;

Sorting the access objects according to the frequency and constructing a statistical histogram;

Based on the statistical histogram, determining a linear relationship between the frequency and the sorting sequence number of the access object;

The characteristic parameter of the user load is determined according to the slope of the linear relationship.

For example, in the real-time operation of the system, the user may generate multiple input/output access requests. According to the information included in the access request, the user input/output access object and frequency within the preset time window can be determined. As shown in Figure 5, within the preset time window, in the order from early to late, the access objects can be ABCDEBDC, then the access objects can include A, B, C, D, E, and the frequency of each access object can be access A frequency: 1, access B frequency: 2, access C frequency: 2, access D frequency: 2, access E frequency: 1. The preset time window can be set according to user needs, and this application does not limit the setting method of the preset time window.

By sorting the access objects according to the frequency, a statistical histogram can be constructed. The horizontal axis of the statistical histogram can be the sorting number of the access object, and the vertical axis can be the value of the frequency. Referring to the above example, in order from high to low, the order of the access objects is B, C, D (all 2 times), A, E (all 1 time), where the order between B, C, D can be randomly determined, and the order between A and E can be randomly determined. The horizontal axis of the obtained statistical histogram can be B, C, D, A, E, and the vertical axis can be the access frequency corresponding to these access objects. Based on the statistical histogram, the linear relationship between the frequency and the sorting number of the access object can be determined. For example, the logarithm of the frequency and the sorting number of the access object can be taken respectively, and then the linear relationship of the logarithm can be determined. For example, the slope α of the linear relationship can be calculated by fitting the least squares method y=xα, where the logarithm of the frequency and the sorting number of the access object are substituted into y and x respectively, and the slope α of the obtained linear relationship is used as the characteristic parameter of the user load.

Those skilled in the art should understand that determining the linear relationship between the frequency and the sorting number of the access object based on the statistical histogram, as well as determining the slope of the linear relationship, can also be achieved by other methods in the prior art, and this application does not limit this.

The following describes an exemplary method for adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object (step S42) according to characteristic parameters of the user load and type parameters of the garbage collection algorithm according to an embodiment of the present application.

In a possible implementation, step S42 includes:

From among multiple preset configurations of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object, selecting a configuration that matches the sum of the available reserved space of the first object and the available reserved space of the second object, the characteristic parameter, and the type parameter as the optimal configuration;

According to the optimal configuration mode, the size of the actual reserved space of the first object and the size of the actual reserved space of the second object are adjusted.

For example, there may be multiple combinations of the sum of space, characteristic parameters, and type parameters, and multiple configuration modes of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object may be pre-set, so that the pre-set configuration mode includes the configuration mode of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object that achieves the best performance of the storage system under each combination of the sum of space, characteristic parameters, and type parameters. That is, the configuration mode that matches the sum of space, characteristic parameters, and type parameters of the size of the available reserved space of the first object and the size of the available reserved space of the second object may be the best configuration mode of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.

For methods and examples of obtaining various configuration methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object, please refer to the relevant description below and Table 1. For example, referring to Table 1, when the space sum is R1, the characteristic parameter is α1, and the type parameter is GC Algor1, the configuration method that matches the space sum, the characteristic parameter, and the type parameter can be M1; N1. The size of the actual reserved space of the first object is M1, and the size of the actual reserved space of the second object is N1. When adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object, adjust the size of the actual reserved space of the first object to M1, and adjust the size of the actual reserved space of the second object to N1. Its exemplary implementation method can be referred to the relevant description below and Figure 8.

The following describes an exemplary method for determining the size of the actual reserved space of the first object and the size of the actual reserved space of the second object in various configuration modes according to an embodiment of the present application.

In a possible implementation, the method further includes:

Determine a value range of the total space according to the hot backup space, the initial reserved space, the user visible space, and the available reserved space of the second object;

For each value in the value range of the space sum, determine multiple division methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object under the value;

According to the multiple characteristic parameters of the user load, the multiple types of parameters of the garbage collection algorithm, the total space and the multiple division methods, multiple configuration methods of the actual reserved space size of the first object and the actual reserved space size of the second object are determined.

For example, since the total space indicates the sum of the available reserved space of the first object and the second object, and the available reserved space of the first object includes the hot backup space at the system layer, the initial reserved space, and the unused space in the user-visible space, wherein the hot backup space and the initial reserved space are fixed, the maximum value of the unused space in the user-visible space is the user-visible space size, and the minimum value of the unused space in the user-visible space is 0, then the value range of the total space can be determined based on the hot backup space, the initial reserved space, the user-visible space, and the available reserved space of the second object.

For each value in the range of the space sum, multiple ways of dividing the size of the actual reserved space of the first object and the size of the actual reserved space of the second object under the value can be determined. That is, multiple combinations of M and N that satisfy "M+N＝R; R∈[r1,r2]", where M represents the size of the actual reserved space of the first object, N represents the size of the actual reserved space of the second object, R represents the space sum, r1, r2 represent the minimum and maximum values of the space sum, respectively, and R∈[r1,r2] represents the value range of the space sum.

The user load can have multiple characteristic parameters, and the garbage collection algorithm can have multiple types of parameters. The storage system performance is related to the characteristic parameters of the user load, the type parameters of the garbage collection algorithm, the size of the actual reserved space of the first object, and the size of the actual reserved space of the second object. Therefore, based on the multiple characteristic parameters of the user load, the multiple types of parameters of the garbage collection algorithm, the total space, and multiple division methods, all possible resource allocation methods of the storage system can be enumerated, and the system performance corresponding to each resource allocation method can be determined. The resource allocation method with the best performance can be selected to determine the configuration method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.

In a possible implementation, the multiple configuration methods of determining the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the multiple characteristic parameters of the user load, the multiple type parameters of the garbage collection algorithm, the total space, and the multiple division methods include:

Determine multiple combinations of the characteristic parameters, the type parameters, and the space sum according to the multiple characteristic parameters of the user load, the multiple type parameters of the garbage collection algorithm, and the space sum, wherein at least one of the characteristic parameters, the type parameters, and the space sum in different combinations is different;

For each combination of the characteristic parameter, the type parameter, and the total space, respectively determine a write amplification factor for each division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object;

A division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object corresponding to the minimum write amplification factor in each combination method is determined as a configuration method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.

For example, based on various characteristic parameters of user load, various type parameters of garbage collection algorithm, and the total space, various combinations of characteristic parameters, type parameters, and the total space can be determined. For any combination of characteristic parameters, type parameters, and the total space, when different combinations of the size M of the actual reserved space of the first object and the size N of the actual reserved space of the second object are used, the system performance may be different. The combination of M and N with the best performance can be screened out as the optimal configuration method of the size M of the actual reserved space of the first object and the size N of the actual reserved space of the second object under the combination of characteristic parameters, type parameters, and the total space. By traversing all combinations of characteristic parameters, type parameters, and the total space, the optimal configuration method of the size M of the actual reserved space of the first object and the size N of the actual reserved space of the second object under each combination is found, thereby realizing multiple configuration methods of determining the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.

The performance of the system can be reflected by the write amplification factor (WAF). The write amplification factor is a parameter that characterizes the degree of write amplification. The determination method of the write amplification factor can be implemented based on the existing technology and will not be repeated here.

For a single object, the write amplification factor has a logarithmic function-like relationship with the garbage collection algorithm, the reserved space size, the user load, etc. FIG6 shows an example of a functional relationship between the write amplification factor and the reserved space size of a single object according to an embodiment of the present application. As shown in FIG6, the larger the reserved space, the smaller the corresponding write amplification factor, indicating that the object performance is better.

If the network connection between objects is considered, the write amplification factor at the system level and the reserved space size of the two objects can present a convex function relationship. Figure 7 shows an example of the functional relationship between the write amplification factor of the storage system according to an embodiment of the present application and the reserved space size of the two objects. As shown in Figure 7, under the premise that the characteristic parameters, type parameters, and the total reserved space R are certain, the reserved space size of the two objects corresponding to the minimum value of the write amplification factor (i.e., the size M of the actual reserved space of the first object and the size N of the actual reserved space of the second object) can be the optimal configuration method under the combination of the characteristic parameters, type parameters, and the total reserved space. Table 1 shows an example of multiple configuration methods for the size of the actual reserved space of the first object and the size of the actual reserved space of the second object determined in an embodiment of the present application.

Table 1

In the example of Table 1, the characteristic parameters of the user load include α1, etc., the space sum includes R1 and R2, etc., and the type parameters of the garbage collection algorithm include GC Algor1, GC Algor2, etc. Then, under the combination of the characteristic parameter α1, the type parameter GC Algor1, and the reserved space sum R1, the optimal configuration mode of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object may be M1; N1, where M1 may indicate the size of the actual reserved space of the first object, N1 may indicate the size of the actual reserved space of the second object, and M1+N1=R1.

Similarly, under the combination of characteristic parameter α1, type parameter GC Algor1, and total reserved space R2, the optimal configuration method of the actual reserved space size of the first object and the actual reserved space size of the second object can be M2; N2, under the combination of characteristic parameter α1, type parameter GC Algor2, and total reserved space R1, the optimal configuration method of the actual reserved space size of the first object and the actual reserved space size of the second object can be M3; N3, under the combination of characteristic parameter α1, type parameter GC Algor2, and total reserved space R2, the optimal configuration method of the actual reserved space size of the first object and the actual reserved space size of the second object can be M4; N4.

Those skilled in the art should understand that in Table 1, the optimal configuration of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object can also be reflected by using other features. For example, the value of the size of the actual reserved space of the first object is M1, and the value of the size of the actual reserved space of the second object is N1. The proportional relationship M1:N1 between the two can be determined based on M1 and N1, and M1:N1 can be stored in the table as a configuration method. This application does not limit the specific storage method of the configuration method.

Those skilled in the art should understand that the quality of system performance can also be characterized by using other parameters besides the write amplification factor, and the present application does not limit the specific method of combining the size of the actual reserved space of the first object with the best screening performance and the size of the actual reserved space of the second object.

In this way, multiple configurations of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object can be determined. The multiple configurations finally determined are all optimal in performance, thus saving the cost of storing multiple configurations.

The following describes an exemplary method for adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode in an embodiment of the present application.

Fig. 8 shows a schematic diagram of adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to an embodiment of the present application. In the example of Fig. 8, assuming that the first object includes a system layer, the second object includes a device layer, and the size of the user available space of the system layer and the size of the unused space in the disk array are both 0, then the size of the available reserved space of the system layer (i.e., the available reserved space of the first object) is equal to the sum of the hot backup space and the initial reserved space (reserved space 1) of the system layer, the size of the available reserved space of the device layer (i.e., the available reserved space of the second object) is equal to the initial reserved space (reserved space 2) of the device layer, and the total space is equal to the sum of the hot backup space, the initial reserved space (reserved space 1) of the system layer, and the initial reserved space (reserved space 2) of the device layer. The left half of the dotted line represents the size of the actual reserved space at the system layer (i.e., the actual reserved space of the first object), and the right half of the dotted line represents the size of the actual reserved space at the device layer (i.e., the actual reserved space of the second object). If the dotted line moves to the left, it means that the size of the actual reserved space of the first object increases, while the size of the actual reserved space of the second object decreases; if the dotted line moves to the right, it means that the size of the actual reserved space of the first object decreases, while the size of the actual reserved space of the second object increases.

In a possible implementation, in the optimal configuration mode, when the size of the actual reserved space of the first object decreases and the size of the actual reserved space of the second object increases, adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode includes at least one of the following:

Lowering a watermark parameter of the garbage collection algorithm, wherein the garbage collection algorithm is started when the value of the watermark parameter is greater than the size of the actual reserved space of the first object;

increasing a speed parameter of the garbage collection algorithm, the speed parameter indicating a garbage collection speed when the storage system executes the garbage collection algorithm;

Reduce the startup delay of the garbage collection algorithm.

For example, if the size of the actual reserved space of the first object indicated by the optimal configuration is smaller than the size of the actual reserved space of the first object at the current moment (i.e., the size of the actual reserved space of the first object is reduced, and the size of the actual reserved space of the second object is increased), it can be considered that less actual reserved space needs to be allocated to the first object. In other words, the garbage collection process needs to be accelerated for the first object. In this case, the actual reserved space of the first object and the second object can be adjusted in at least one of the following ways.

Method one may be to lower the watermark parameter of the garbage collection algorithm. The watermark parameter indicates the maximum capacity of the first object to store data. Therefore, when the size of the actual reserved space of the first object is greater than the value of the watermark parameter, it can be considered that the first object still has margin for new data to be written; when the size of the actual reserved space of the first object is equal to the value of the watermark parameter, it can be considered that the margin of the first object is exactly 0; when the size of the actual reserved space of the first object is less than the value of the watermark parameter, the data needs to be erased, that is, the garbage collection algorithm needs to be started when the value of the watermark parameter is greater than the size of the actual reserved space of the first object. Once the garbage collection algorithm is started, garbage collection can be performed, so that the size of the actual reserved space of the first object gradually increases until it reaches the size of the actual reserved space of the first object indicated by the optimal configuration method.

The second method may be to increase the speed parameter of the garbage collection algorithm. The speed parameter indicates the garbage collection speed when the storage system executes the garbage collection algorithm. Therefore, the larger the speed parameter, the faster the garbage collection speed, that is, accelerated data erasure is achieved, so that the storage system, especially the first object, can adapt to the new configuration mode as soon as possible.

The third method is to reduce the startup delay of the garbage collection algorithm. There may be a certain startup delay from receiving the startup instruction to the actual startup of the garbage collection algorithm. By reducing this startup delay, garbage collection can also be accelerated, so that the storage system, especially the first object, can adapt to the new configuration mode as soon as possible.

For any of the above methods, the second object can adaptively adjust the size of the actual reserved space of the second object based on the existing technology, which will not be described in detail here.

The above method does not require changing the size of the hot backup space, the initial reserved space, etc., but instead adjusts the actual reserved space size of the first object and the second object by adjusting the waterline parameters, speed parameters, or startup delay. In other words, the space (actual reserved space) that can be used for garbage collection of the first object and the second object in actual use is adjusted.

In a possible implementation, under the optimal configuration, when the size of the actual reserved space of the first object increases and the size of the actual reserved space of the second object decreases, adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration includes at least one of the following:

Increasing a watermark parameter of the garbage collection algorithm, wherein the garbage collection algorithm is started when the value of the watermark parameter is greater than the size of the actual reserved space of the first object;

reducing a speed parameter of the garbage collection algorithm, the speed parameter indicating a garbage collection speed when the storage system executes the garbage collection algorithm;

Increase the startup delay of the garbage collection algorithm.

For example, after determining the optimal configuration, the actual reserved space of the first object and the second object can be adjusted according to the size of the actual reserved space of the first object indicated by the optimal configuration and the size of the actual reserved space of the second object. If the size of the actual reserved space of the first object indicated by the optimal configuration is larger than the size of the actual reserved space of the first object at the current moment (that is, the size of the actual reserved space of the first object increases, and the size of the actual reserved space of the second object decreases), it can be considered that more reserved space resources need to be allocated to the first object. That is, for the first object, the garbage collection process needs to be slowed down. In this case, at least one of the following methods can be used to adjust the actual reserved space of the first object and the second object.

Method 1 may be to increase the waterline parameter of the garbage collection algorithm. As can be seen from the above description, the garbage collection algorithm needs to be started when the value of the waterline parameter is greater than the size of the actual reserved space of the first object, and the size of the actual reserved space of the first object indicated by the optimal configuration method is larger than the size of the actual reserved space of the first object at the current moment, so the garbage collection algorithm can be started later. In this case, the waterline parameter of the garbage collection algorithm can be increased so that the garbage collection algorithm can be started later, thereby reducing the data processing cost of running the garbage collection algorithm.

The second method is to reduce the speed parameter of the garbage collection algorithm. The speed parameter indicates the garbage collection speed when the storage system executes the garbage collection algorithm. Therefore, the smaller the speed parameter, the slower the garbage collection speed, so that the storage system, especially the first object, can adapt to the new configuration mode as soon as possible.

The third method is to increase the startup delay of the garbage collection algorithm. There may be a certain startup delay from receiving the startup instruction to the actual startup of the garbage collection algorithm. By increasing this startup delay, garbage collection can also be slowed down, so that the storage system, especially the first object, can adapt to the new configuration mode as soon as possible.

An embodiment of the present application provides a resource allocation device. FIG9 is a schematic diagram showing the structure of the resource allocation device according to the embodiment of the present application.

As shown in FIG9 , in a possible implementation manner, an embodiment of the present application provides a resource allocation device, the device comprising:

An acquisition module 91 is used to acquire the size of the available reserved space of the first object and the size of the available reserved space of the second object;

The adjustment module 92 is used to adjust the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm.

In a possible implementation manner, the device further includes:

A first determination module is used to determine the user input/output access object and frequency within a preset time window;

A construction module, used to sort the access objects according to the frequency and construct a statistical histogram;

A second determination module, configured to determine a linear relationship between the frequency and the sorting sequence number of the access object based on the statistical histogram;

The third determination module is used to determine the characteristic parameter of the user load according to the slope of the linear relationship.

In a possible implementation, the adjustment module includes:

a selecting unit, configured to select, from among a plurality of preset configuration modes of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object, a configuration mode that matches the sum of the space of the available reserved space of the first object and the size of the available reserved space of the second object, the characteristic parameter, and the type parameter as the optimal configuration mode;

An adjusting unit is used to adjust the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode.

In a possible implementation manner, the device further includes:

a fourth determining module, configured to determine a value range of the total space according to the hot backup space, the initial reserved space, the user visible space, and the available reserved space of the second object;

a fifth determining module, configured to determine, for each value in the value range of the space sum, a plurality of division methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object under the value;

The sixth determination module is used to determine multiple configuration methods of the actual reserved space size of the first object and the actual reserved space size of the second object based on multiple characteristic parameters of the user load, multiple types of parameters of the garbage collection algorithm, the total space and the multiple division methods.

In a possible implementation manner, the sixth determining module includes:

a first determining unit, configured to determine, according to the plurality of characteristic parameters of the user load, the plurality of type parameters of the garbage collection algorithm, and the space sum, a plurality of combinations of the characteristic parameters, the type parameters, and the space sum, wherein at least one of the characteristic parameters, the type parameters, and the space sum in different combinations is different;

a second determining unit, configured to determine, for each combination of the characteristic parameter, the type parameter, and the space sum, a write amplification factor for each division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object;

The third determination unit is used to determine the division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object corresponding to the minimum write amplification factor under each combination method as a configuration method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.

Reduce the startup delay of the garbage collection algorithm.

Increase the startup delay of the garbage collection algorithm.

An embodiment of the present application provides a resource allocation device, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.

An embodiment of the present application provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

An embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

As shown in FIG10 , the resource allocation device may include at least one of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, or a smart city device, and a server device. The embodiment of the present application does not impose any special restrictions on the specific type of the resource allocation device.

The resource allocation device may include a processor 110, a memory 121, and a communication module 160. It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the resource allocation device. In other embodiments of the present application, the resource allocation device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated in one or more processors.

The processor can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. The memory may store instructions or data that have been used or are frequently used by the processor 110, such as the multiple configuration modes of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object pre-set in the embodiment of the present application. If the processor 110 needs to use the instruction or data, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

The memory 121 can be used to store computer executable program codes, which include instructions. The memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, an application required for at least one function (such as the least squares method), etc. The data storage area can store data acquired or created during the use of the resource allocation device (such as the size of the available reserved space of the first object and the size of the available reserved space of the second object, etc.). In addition, the memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional methods of the resource allocation device or the above-mentioned resource allocation method by running instructions stored in the memory 121 and/or instructions stored in a memory provided in the processor.

The communication module 160 can be used to receive data (such as the characteristic attributes in the embodiment of the present application) from other devices or equipment through wireless communication/wired communication, and output data to other devices or equipment. For example, wireless communication solutions including WLAN (such as Wi-Fi network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. can be provided.

A computer readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing.

The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing/processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device.

The computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by utilizing the status information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions to implement various aspects of the present application.

Various aspects of the present application are described herein with reference to the flowcharts and/or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and/or block diagram and the combination of each box in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, system, method and computer program product according to multiple embodiments of the present application. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved.

It should also be noted that each box in the block diagram and/or flowchart, and the combination of boxes in the block diagram and/or flowchart, can be implemented by hardware (such as circuits or ASICs (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.

Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

A resource allocation method, characterized in that the method comprises:

Get the size of the available reserved space of the first object and the size of the available reserved space of the second object;

The size of the actual reserved space of the first object and the size of the actual reserved space of the second object are adjusted according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm.
The method according to claim 1, wherein the first object and the second object are set in a storage system, and the storage system runs the garbage collection algorithm.

The first object includes a system layer of the storage system, and the second object includes a device layer of the storage system.

The available reserved space of the first object includes the hot backup space of the system layer, the initial reserved space, and the unused space in the user visible space;

The available reserved space of the second object includes the initial reserved space of the device layer;

Among them, the hot backup space and the initial reserved space of the system layer and the initial reserved space of the device layer remain unchanged.
The method according to claim 1, wherein the first object and the second object are set in a storage system, and the storage system runs the garbage collection algorithm.

The first object includes a system layer of the storage system, and the second object includes an application layer of the storage system.

The available reserved space of the first object includes the hot backup space of the system layer, the initial reserved space, and the unused space in the user visible space;

The available reserved space of the second object includes the initial reserved space of the application layer;

Among them, the hot backup space and the initial reserved space of the system layer remain unchanged.
The method according to any one of claims 1 to 3, characterized in that before adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm, the method further comprises:

Determine the user input/output access objects and frequencies within a preset time window;

Sorting the access objects according to the frequency and constructing a statistical histogram;

Based on the statistical histogram, determining a linear relationship between the frequency and the sorting sequence number of the access object;

The characteristic parameter of the user load is determined according to the slope of the linear relationship.
The method according to any one of claims 1 to 4, characterized in that the step of adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to a characteristic parameter of a user load and a type parameter of a garbage collection algorithm comprises:

From among multiple preset configurations of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object, selecting a configuration that matches the sum of the available reserved space of the first object and the available reserved space of the second object, the characteristic parameter, and the type parameter as the optimal configuration;

According to the optimal configuration mode, the size of the actual reserved space of the first object and the size of the actual reserved space of the second object are adjusted.
The method according to claim 5, characterized in that the method further comprises:

Determine a value range of the total space according to the hot backup space, the initial reserved space, the user visible space, and the available reserved space of the second object;

For each value in the value range of the space sum, determine multiple division methods of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object under the value;

According to the multiple characteristic parameters of the user load, the multiple types of parameters of the garbage collection algorithm, the total space and the multiple division methods, multiple configuration methods of the actual reserved space size of the first object and the actual reserved space size of the second object are determined.
The method according to claim 6, characterized in that the multiple configuration methods of determining the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the multiple characteristic parameters of the user load, the multiple type parameters of the garbage collection algorithm, the total space, and the multiple division methods include:

Determine multiple combinations of the characteristic parameters, the type parameters, and the space sum according to the multiple characteristic parameters of the user load, the multiple type parameters of the garbage collection algorithm, and the space sum, wherein at least one of the characteristic parameters, the type parameters, and the space sum in different combinations is different;

For each combination of the characteristic parameter, the type parameter, and the total space, respectively determine a write amplification factor for each division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object;

A division method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object corresponding to the minimum write amplification factor in each combination method is determined as a configuration method of the size of the actual reserved space of the first object and the size of the actual reserved space of the second object.
The method according to any one of claims 5 to 7, characterized in that, in the optimal configuration mode, when the size of the actual reserved space of the first object decreases and the size of the actual reserved space of the second object increases,

The adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode includes at least one of the following:

Lowering a watermark parameter of the garbage collection algorithm, wherein the garbage collection algorithm is started when the value of the watermark parameter is greater than the size of the actual reserved space of the first object;

increasing a speed parameter of the garbage collection algorithm, the speed parameter indicating a garbage collection speed when the storage system executes the garbage collection algorithm;

Reduce the startup delay of the garbage collection algorithm.
The method according to any one of claims 5 to 8, characterized in that, in the optimal configuration mode, when the size of the actual reserved space of the first object increases and the size of the actual reserved space of the second object decreases,

The adjusting the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the optimal configuration mode includes at least one of the following:

Increasing a watermark parameter of the garbage collection algorithm, wherein the garbage collection algorithm is started when the value of the watermark parameter is greater than the size of the actual reserved space of the first object;

reducing a speed parameter of the garbage collection algorithm, the speed parameter indicating a garbage collection speed when the storage system executes the garbage collection algorithm;

Increase the startup delay of the garbage collection algorithm.
A resource allocation device, characterized in that the device comprises:

An acquisition module, configured to acquire the size of the available reserved space of the first object and the size of the available reserved space of the second object;

The adjustment module is used to adjust the size of the actual reserved space of the first object and the size of the actual reserved space of the second object according to the characteristic parameters of the user load and the type parameters of the garbage collection algorithm.
A resource allocation device, characterized by comprising:

processor;

a memory for storing processor-executable instructions;

Wherein, the processor is configured to implement the method according to any one of claims 1 to 9 when executing the instructions.
A non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method of any one of claims 1 to 9.
A computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, characterized in that when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method described in any one of claims 1 to 9.