CN108459825B - Storage system service quality control method and device - Google Patents
Storage system service quality control method and device Download PDFInfo
- Publication number
- CN108459825B CN108459825B CN201810089640.2A CN201810089640A CN108459825B CN 108459825 B CN108459825 B CN 108459825B CN 201810089640 A CN201810089640 A CN 201810089640A CN 108459825 B CN108459825 B CN 108459825B
- Authority
- CN
- China
- Prior art keywords
- volume
- waiting
- queue
- input
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention provides a method and a device for controlling the service quality of a storage system. The method comprises the following steps: when an idle thread exists in the thread pool and the waiting queue is not empty, determining a volume to be allocated according to the frequency of executing input/output operation of each volume in the waiting queue; an idle thread is assigned to a volume to be allocated such that the volume to be allocated performs all pending input/output operations belonging to that volume. The device comprises: the device determining module is used for determining the volumes to be allocated according to the frequency of executing input/output operations of the volumes in the waiting queue when the thread pool has an idle thread and the waiting queue is not empty; and the thread allocation module is used for allocating idle threads to the volume to be allocated so that the volume to be allocated executes all the input/output operations which belong to the volume and are waiting to be executed. The method and the device for controlling the service quality of the storage system can give consideration to fairness and continuity and improve the service quality of the storage system.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a method and an apparatus for controlling quality of service of a storage system.
Background
When data on a host needs to be accessed to a data object on the host, for example, a data address or a file corresponding to an application program, the host accesses the storage system to obtain data corresponding to the object in the storage system. When the storage system processes an access request of the host, certain system resources are allocated for the current access, and the system resources of the storage system include, for example, a memory, a CPU, and the like. The process of adjusting the access Service performance by adjusting the occupied system resources is called Quality of Service (QoS) control.
QoS control, as a network security mechanism of a network, is a technique for solving the problems of network delay and congestion, by providing better service capability for specified network communication.
For network traffic, the quality of service includes transmission bandwidth, transmission delay, packet loss rate of data, and the like. In the network, the service quality can be improved by ensuring the transmission bandwidth, reducing the transmission time delay, reducing the packet loss rate of data, reducing the time delay jitter and other measures.
Storage domain QoS is the mechanism that regulates the storage IOPS consumption by means of QoS management functions. The I/O delivery speed is increased by managing performance by allocating physical resources to individual workloads. The method does not improve the overall processing capacity of the system, and is only responsible for reasonable allocation of resources. It includes control of bandwidth, reservations of the IOPS, upper bounds and priority, etc.
The QoS is mainly measured by an upper and lower limit of IOPS (Input/Output Operations Per Second), an upper and lower limit of MBPS (milli bits Per Second), a weight share, and a delay upper limit. In a storage system, a contention mode is usually adopted to determine the order in which each application I/O (Input/Output) is issued to hardware, so that the service quality of some application I/O or some application I/O cannot be guaranteed. The storage QoS reasonably distributes the resources of the storage system under the condition of ensuring the IOPS, MBPS or time delay of the applications, so that the performance of a single or certain applications is improved while the resources among the applications are isolated, and the service quality is further ensured.
The traditional storage QoS method takes a pool as granularity, the flow is limited by independently setting an IOPS for each volume on an interface through a user, and if the flow is not limited externally, the situation that a certain disk always preempts resources can occur. But doing so at too great a granularity can impact QoS functionality.
Therefore, in order to ensure fairness, the storage QoS control method in the prior art mainly includes that a storage QoS service layer uses a single I/O of each volume (application) as a granularity, arranges an issuing sequence according to a preset algorithm, for example, SFQ (Start-time fair queuing), and the like, and the algorithm is that the storage QoS service layer calculates a timestamp of each I/O according to a set IOPS lower limit, MBPS lower limit, weight, and the like, marks the timestamp, sorts each I/O according to the sequence of the timestamps, and issues the timestamp to bottom hardware according to the ordered sequence. This is done at a slow processing speed. In the prior art, the user needs to set the QoS value or priority of a certain volume to ensure the performance of the volume. If the flow is not limited independently, a certain volume can always seize the flow, and the fairness cannot be ensured. In addition, in the prior art, I/O is taken as a scheduling unit for guaranteeing fairness. However, if continuous reading and writing are required for a certain area, this approach may prolong the processing time and destroy the continuity of the volume, resulting in poor overall performance and service quality of the storage system.
Disclosure of Invention
In order to overcome the defect that the prior art cannot give consideration to fairness and continuity, the invention provides a method and a device for controlling the service quality of a storage system.
According to an aspect of the present invention, there is provided a storage system quality of service control method, including:
s1, when an idle thread exists in the thread pool and the waiting queue is not empty, determining the volume to be allocated according to the frequency of executing input/output operation of each volume in the waiting queue, and adding the volume to be allocated into the execution queue;
and S2, allocating an idle thread to the volume to be allocated, so that the volume to be allocated executes all input/output operations which belong to the volume and are waiting to be executed.
Preferably, the step S1 further includes:
s11, when an idle thread exists in the thread pool, judging whether the waiting queue is empty;
s12, when the waiting queue is empty, storing the volume to which the input/output operation waiting to be executed belongs into the waiting queue;
s13, according to the frequency of executing input/output operation of each volume in the waiting queue, determining the volume to be distributed, deleting the volume to be distributed from the waiting queue, and adding the volume to be distributed to the executing queue.
Preferably, the state of the volume includes: a waiting processing state, a processing state, an execution ending state and a waiting destroying state;
when the volume is in the waiting queue, the state of the volume is a waiting processing state;
when the volume does not finish all input/output operations waiting to be executed and the number of times of executing the input/output operations does not reach the preset maximum number, the state of the volume is a processing state;
when the volume does not finish all input/output operations waiting to be executed and the number of times of executing the input/output operations reaches the preset maximum number, the state of the volume is an execution ending state;
when the volume finishes executing all input/output operations waiting to be executed, the state of the volume is a state waiting to be destroyed.
Preferably, the step S12 is followed by:
and deleting the volume in the execution queue in the state of waiting for destruction, and releasing the resources occupied by the volume.
Preferably, the specific step of determining the volume to be allocated according to the frequency of executing the input/output operation of each volume in the waiting queue includes:
sequentially calculating the average value of the input/output operation times per second of each volume of the waiting queue according to the sequence from the head of the queue to the tail of the queue until the average value is smaller than a preset threshold value, and determining the volume of which the average value is smaller than the preset threshold value as the volume to be distributed;
wherein the average of the number of times of performing the input/output operation per second is an average of the number of times of performing the input/output operation per second of the volume from when the input/output request starts to be processed to when the average is calculated.
Preferably, the step S12 specifically includes:
when the volume to which the input/output operation waiting to be executed belongs is not in the waiting queue and is not in the execution queue, adding the volume to the tail of the waiting queue;
when the volume to which the input/output operation waiting to be executed belongs is in the waiting queue, keeping the volume at the original position in the waiting queue;
when a volume to which an input/output operation waiting for execution belongs is in an execution queue, judging the state of the volume; and when the state of the volume is an execution ending state or a destruction waiting state, deleting the volume from the execution queue and adding the volume to the tail of the waiting queue.
Preferably, the step S12 further includes:
when the waiting queue is not empty, the process proceeds to step S13.
Preferably, the step S2 is followed by:
s3, when the volume in any thread finishes executing all the I/O operations waiting to be executed, or the number of executing I/O operations reaches the preset maximum number, releasing the occupied thread resource, making the thread become an idle thread, and returning to execute step S1.
According to another aspect of the present invention, there is provided a storage system quality of service control apparatus comprising:
the device determining module is used for determining the volume to be allocated according to the frequency of executing input/output operations of each volume in the waiting queue when an idle thread exists in the thread pool and the waiting queue is not empty, and adding the volume to be allocated to the execution queue;
and the thread allocation module is used for allocating an idle thread to the volume to be allocated so that the volume to be allocated executes all the input/output operations which belong to the volume and are waiting to be executed.
Preferably, the device determination module comprises:
the judging submodule judges whether the waiting queue is empty or not when an idle thread exists in the thread pool;
the queue submodule is used for storing the volume to which the input/output operation to be executed belongs into the waiting queue when the waiting queue is empty;
the selection submodule determines a volume to be allocated according to the frequency of executing input/output operation of each volume in the waiting queue, deletes the volume to be allocated from the waiting queue and adds the volume to be allocated to the execution queue;
and the recovery submodule is used for deleting the volume in the execution queue in the state of waiting for destruction and releasing the resources occupied by the volume.
The invention provides a method and a device for controlling the service quality of a storage system, which determine a volume to be allocated according to the frequency of executing input/output operations of each volume in a waiting queue, allocate an idle thread to the volume to be allocated, and enable the volume to be allocated to execute all the input/output operations to be executed belonging to the volume, so that the fairness and the continuity can be considered when the service quality of the storage system is controlled under the condition that a plurality of volumes are subjected to the input/output operations to be executed, and the service quality of the storage system is improved.
Drawings
FIG. 1 is a flow chart of a method for controlling the quality of service of a storage system according to an embodiment of the present invention;
fig. 2 is a functional block diagram of a storage system qos control apparatus according to an embodiment of the present invention.
Detailed Description
The following detailed description of embodiments of the present invention is provided in connection with the accompanying drawings and examples. The following examples are intended to illustrate the invention but are not intended to limit the scope of the invention.
Fig. 1 is a flowchart of a method for controlling a quality of service of a storage system according to an embodiment of the present invention. As shown in fig. 1, a method for controlling the quality of service of a storage system includes: step S1, when there is idle thread in the thread pool and the waiting queue is not empty, according to the frequency of executing input/output operation of each volume in the waiting queue, determining the volume to be allocated, and adding the volume to be allocated to the execution queue; step S2, the idle thread is assigned to the volume to be allocated, so that the volume to be allocated performs all pending input/output operations belonging to the volume.
Note that a volume in the embodiment of the present invention refers to a logical volume.
A Logical Volume (Logical Volume) is a virtual disk formed by Logical disks, and may also be referred to as a disk partition, and may be simply referred to as a Volume. The volume is a disk management mode of a Windows system, and aims to jump the hard disk space from the management mode of a physical hard disk so as to carry out more convenient unified management and distribution. For example, when there is an 8GB hard disk and a 20GB hard disk, it is desired to divide the disks into two logical disks, 12GB and 16GB, which cannot be done by physical partitioning, but can be done by volume management. Each volume can be regarded as a logical disk, and can be a logical disk of a physical hard disk, i.e. a logical disk represented by the drive symbols of the D disk and the E disk which are directly seen, or a RAID 0 or RAID 1 array composed of two hard disks or partial spaces of two hard disks, or other RAID 5 arrays composed of more hard disks.
Thread pools are a form of multi-threaded processing in which tasks are added to a queue and then automatically started after a thread is created. The thread pool threads are all background threads. Each thread uses a default stack size, runs at a default priority, and is in a multi-threaded unit. If a thread is idle in managed code (e.g., waiting for an event), the thread pool will insert another helper thread to keep all processors busy. If all thread pool threads remain busy all the time, but pending work is contained in the queue, the thread pool will create another helper thread after a period of time but the number of threads never exceeds the maximum.
The thread pool includes a plurality of threads. The thread pool is utilized, so that the threads can be reused, namely, one task is executed, the threads are not destroyed, and other tasks can be continuously executed.
In the thread pool, the thread which finishes one task is an idle thread; the idle thread may be used to continue performing other tasks.
Specifically, in the embodiment of the invention, the storage system utilizes the thread pool to execute the concurrent tasks of multiple threads. In each thread, volumes in the storage system perform input/output operations to complete tasks.
After mapping a volume in a storage system, especially a distributed storage system, a real block device file is associated, and corresponding information of the volume and the device is recorded. Input/output (I/O) tasks occur when a particular location of the volume needs to be read from or written to. Each input/output task includes a certain number of input/output operations. The volume is referred to as a volume to which the certain number of input/output operations belong, and the certain number of input/output operations are referred to as input/output operations belonging to the volume.
The storage system includes two queues: a wait queue and an execute queue.
The volumes are stored in a wait queue for pending I/O operations.
Through certain selection mechanisms, the selected volume in the wait queue is moved from the wait queue to the execute queue.
The idle thread may be assigned to a volume in the execution queue that is not assigned a thread, i.e., to a volume in the execution queue that is not performing an input/output operation.
And step S1, when the thread pool has idle threads and the waiting queue is not empty, determining the volume to be allocated according to the priority of each volume in the waiting queue.
When the waiting queue is not empty, the volume to which the input/output operation waiting to be executed belongs is not allocated with a thread. Thus, when there is a thread that is idle, it is necessary to allocate the thread that is idle so that a volume in the wait queue has an opportunity to perform the pending input/output operations belonging to that volume.
Volumes that have not performed input/output operations for a long time have a higher priority, while volumes that have recently performed input/output operations may have a lower priority.
Therefore, the priority of each volume in the queue is waited, and the frequency of executing input/output operation of each volume is determined.
The frequency of executing I/O operations by a volume refers to the number of I/O operations performed by the volume per unit time within a certain time range.
And according to the frequency of executing input/output operation of each volume, determining the volume to be allocated, and then adding the volume to be allocated to the execution queue.
The volumes to be allocated are determined by the frequency of executing the input/output operation, so that the phenomenon that some volume or volumes always occupy resources to execute the input/output operation and other volumes cannot execute the input/output operation for a long time can be avoided, all volumes are selected and processed fairly, and the balanced scheduling of each volume is ensured.
At step S2, after the volume to be allocated is added to the execution queue, the volume has no volume to perform an input/output operation, and therefore, an idle thread may be allocated to the volume, so that the volume performs all input/output operations to be performed belonging to the volume.
Since the volumes to which the i/o operations belong in the multiple i/o tasks may be the same, in the embodiment of the present invention, when the volume performs the i/o operation after an idle thread is allocated to the volume to be allocated, the i/o operation belonging to the volume in one i/o task is not performed, but all the i/o operations waiting to be performed belonging to the volume in all the i/o tasks are performed.
For example, when there are 3 input/output tasks for each of the volumes 1 in the wait queue, and the number of input/output operations to be performed in the input/output tasks is 20, 30, or 50, respectively, the volume 1 is determined as the volume to be allocated, and after the free thread is allocated to the volume 1, the volume 1 is targeted to perform 100 input/output operations instead of 20, 30, or 50 input/output operations.
In the manner, the volume assigned with the idle thread executes a batch of input/output operations, but not a part of the input/output operations, so that the volume can continuously execute the input/output operations, but does not wait for the idle thread to be assigned again to execute the input/output operations in other tasks after the input/output operations in a certain task are executed, and the processing speed of the storage system is prevented from being greatly reduced due to the fact that the volume assigned with a single input/output operation is repeatedly determined.
The embodiment of the invention determines the volume to be allocated according to the frequency of executing the input/output operation by each volume in the waiting queue, and allocates the idle thread to the volume to be allocated, so that the volume to be allocated executes all the input/output operations to be executed belonging to the volume, and the fairness and the continuity can be considered when the quality of service of the storage system is controlled under the condition that a plurality of volumes have the input/output operations to be executed, and the quality of service of the storage system is improved. Furthermore, the storage system service quality control can be performed more conveniently and quickly without setting the IOPS, MBPS, weight and the like of each volume independently.
Based on the above embodiment, step S1 further includes: step S11, when there is idle thread in the thread pool, judging whether the waiting queue is empty; step S12, when the waiting queue is empty, the volume to which the input/output operation waiting to be executed belongs is saved in the waiting queue; step S13 is to determine the volume to be allocated according to the frequency of executing the input/output operation of each volume in the wait queue, delete the volume to be allocated from the wait queue, and add the volume to the execute queue.
Specifically, step S1 further includes the following steps.
In step S11, when there is an idle thread in the thread pool, it is first determined whether the wait queue is empty.
In step S12, when the wait queue is empty, it indicates that the volumes to which the input/output operations waiting to be executed belong are all allocated with threads. In this case, when there is a newly generated input/output operation waiting for execution, it is necessary to save the volume to which the newly generated input/output operation waiting for execution belongs in the wait queue so that the wait queue is not empty, so that the volume in the wait queue has an opportunity to execute the input/output operation waiting for execution belonging to the volume.
The newly generated input/output operation waiting for execution refers to the input/output operation waiting for execution generated after the volume to which the input/output operation waiting for execution belongs is saved in the waiting queue when the waiting queue is empty last time.
The volume to which the newly generated input/output operation waiting to be executed belongs can be managed by the device management list. When a certain volume has an input/output operation generated, which belongs to the volume, and is waiting to be executed, the volume is saved in a device management list.
In step S13, after the volume to which the newly generated input/output operation waiting to be executed belongs is saved in the wait queue, the wait queue is not empty.
And determining the volume to be allocated according to the frequency of executing input/output operation of each volume in the waiting queue when the waiting queue is not empty.
After the volume to be allocated is determined, the volume to be allocated is deleted from the waiting queue and added into the execution queue, namely, the volume to be allocated is moved from the waiting queue to the execution queue.
And each time the idle thread exists in the thread pool and the waiting queue is empty, acquiring the volume to which the newly generated input/output operation waiting for execution belongs from the equipment management list and storing the volume into the waiting queue, so that the circulation processing can be realized, and all the input/output operations waiting for execution are executed.
According to the embodiment of the invention, the volume to which the newly generated input/output operation to be executed belongs is stored in the waiting queue when the thread pool has an idle thread and the waiting queue is empty, so that the waiting queue is not empty, the input/output task can be circularly processed until all the input/output operations belonging to each volume are executed, and all the input/output operations to be executed can be executed more quickly under the condition of multi-thread concurrence.
Based on the above embodiments, the state of the volume includes: a waiting processing state, a processing state, an execution ending state and a waiting destroying state; when the volume is in the waiting queue, the state of the volume is a waiting processing state; when the volume does not finish all input/output operations waiting to be executed and the number of times of executing the input/output operations does not reach the preset maximum number, the state of the volume is a processing state; when the volume does not finish all input/output operations waiting to be executed and the number of times of executing the input/output operations reaches the preset maximum number, the state of the volume is an execution ending state; when the volume finishes executing all input/output operations waiting to be executed, the state of the volume is a state waiting to be destroyed.
Specifically, in order to better implement the storage system quality of service control, when the storage system quality of service control is performed, the state of the volume includes: a wait for processing state, an in-process state, an execution end state, and a wait for destruction state.
The wait for processing state refers to a state in which the volume waits for the volume determined to be allocated. When the volume is in the waiting queue, the volume waiting is determined to be the volume to be allocated, and the state of the volume is the waiting processing state.
When the number of times of the input/output operations to be executed by a certain volume is too large, the volume always occupies the thread and an idle thread cannot appear, so that the scheduling of the volume is unbalanced, and the effect of controlling the service quality of the storage system cannot be achieved. To avoid this, it is necessary to preset the maximum number of times of performing the input/output operations after each time a thread is assigned to a volume. When the volume performs the input/output operation for a preset maximum number of times after being allocated with the thread, the volume performs the input/output operation forcibly.
The in-process state refers to a state in which the volume is performing an input/output operation. When the volume does not finish executing all input/output operations waiting to be executed and the number of executing the input/output operations does not reach the preset maximum number, the volume is indicated to execute the input/output operations, and the volume state is a processing state.
The execution end state refers to a state in which the volume execution input/output operation is forcibly interrupted. When the volume does not finish executing all input/output operations waiting to be executed and the number of executing the input/output operations reaches the preset maximum number, the volume executes the input/output operations forcibly, and the state of the volume is the execution ending state.
The wait for destroy state refers to a state in which the volume has completed executing all the input/output operations waiting to be executed that belong to the volume. When the volume finishes executing all input/output operations waiting to be executed, the state of the volume is a state waiting to be destroyed.
The current state of each volume can be displayed in real time through a state machine, so that corresponding processing is carried out according to the state of the volume.
Based on the foregoing embodiment, as an optional embodiment, step S12 is followed by: and deleting the volume in the execution queue in the state of waiting for destruction, and releasing the resources occupied by the volume.
Specifically, in step S12, the newly generated volume to which the input/output operation waiting to be executed belongs may or may not be the volume waiting to be destroyed.
When the volume to which the newly generated input/output operation waiting for execution belongs does not include the volume waiting for destruction state, it is described that the new input/output operation belonging to the volume waiting for destruction state is not generated. Thus, in the execution queue, the state of the volume remains as a wait for destroy state. Since there are no pending I/O operations belonging to the volume, the system resources occupied by the volume may be reclaimed in order to more efficiently utilize the system resources.
When the volume to which the newly generated input/output operation waiting for execution belongs includes a volume waiting for destruction, it is described that a new input/output operation belonging to the volume waiting for destruction is generated, and the input/output operation belonging to the volume still waits for execution, and the resources occupied by the volume may not be recycled. After the volume in the new state to be destroyed is saved in the waiting queue, the state of the volume is changed into a waiting processing state.
In order to optimize the multithread concurrency and reduce the lock conflict of the threads, after the volume to which the input/output operation to be executed belongs is stored in the waiting queue in step S12, the volume in the state still waiting for destruction is deleted from the execution queue, and resources such as memory space occupied by the volume are released, so that the resources of the storage system are recycled, and the service quality of the storage system is improved.
The embodiment of the invention recovers the resources occupied by the volume waiting for destruction after storing the volume to which the input/output operation waiting for execution belongs in the waiting queue, thereby more effectively utilizing the system resources and improving the service quality of the storage system.
Based on the foregoing embodiment, as an optional embodiment, the specific step of determining the volume to be allocated according to the frequency of executing the input/output operation of each volume in the waiting queue includes: sequentially calculating the average value of the input/output operation times per second of each volume of the waiting queue according to the sequence from the head of the queue to the tail of the queue until the average value is smaller than a preset threshold value, and determining the volume of which the average value is smaller than the preset threshold value as the volume to be distributed; wherein the average of the number of times of performing the input/output operation per second is an average of the number of times of performing the input/output operation per second from when the input/output operation is started to when the average is calculated.
Specifically, as an alternative embodiment, when determining the volume to be allocated according to the frequency of executing the input/output operation of each volume in the waiting queue, the following steps may be adopted.
The frequency of performing input/output operations for each volume may be measured by an average of the number of times input/output operations per second (IOPS) performed for each volume over a period of time.
For each volume, the average of the number of times of input/output operations (IOPS) per second for each volume is the average of the IOPS for that volume from when the volume starts to perform input/output operations to when the average is calculated.
Since a volume may be assigned threads multiple times during the operation of the storage system, the time when the volume starts performing input/output operations refers to the time when the volume starts performing input/output operations after the first time the volume is assigned a thread.
And calculating the IOPS average value of the first volume from the head of the waiting queue, and comparing the IOPS average value of the volume with a preset threshold value.
When the average IOPS value of the volume is smaller than the preset threshold, it indicates that the volume performs fewer input/output operations within the time period for calculating the average IOPS value, that is, the frequency is low, and therefore, the priority of the volume is high, and the volume can be determined as the volume to be allocated.
When the average IOPS value of the volume is greater than the preset threshold, it indicates that the volume performs input/output operations more frequently, and therefore, the priority of the volume is lower.
And if the IOPS average value of the first volume from the head of the queue of the waiting queue is greater than a preset threshold value, calculating the IOPS average value of the next volume according to the sequence from the head of the queue to the tail of the queue, and comparing the IOPS average value with the preset threshold value.
Through the process, the average value of the number of times of executing the input/output operation per second of each volume of the waiting queue is calculated in sequence until the average value of the number of times of executing the input/output operation per second is smaller than a preset threshold value, and the volume of which the average value of the number of times of executing the input/output operation per second is smaller than the preset threshold value is determined as the volume to be distributed.
If the average value of the number of times of performing the input/output operation per second of each volume in the waiting queue is greater than the preset threshold, the first volume from the head of the waiting queue may be determined as the volume to be allocated.
By the method, the frequency of executing the input/output operation by each volume in the waiting queue does not need to be calculated, the volumes to be allocated can be quickly determined on the premise of ensuring better fairness, and the speed of executing the complete input/output operation by the storage system is increased.
As an alternative embodiment, the frequency of executing input/output operations of each volume in the waiting queue can be calculated, and when there are several idle threads, several volumes with the lowest frequency are determined as the volumes to be allocated.
The embodiment of the invention determines the volumes to be allocated according to the frequency of executing the input/output operation, can avoid the phenomenon that some volume or volumes always occupy resources to execute the input/output operation, and other volumes cannot execute the input/output operation for a long time, ensures that all volumes can be selected and processed fairly, ensures the balanced scheduling of each volume, and improves the service quality of a storage system.
Based on the above embodiment, step S12 specifically includes: when the volume to which the input/output operation waiting to be executed belongs is not in the waiting queue and is not in the execution queue, adding the volume to the tail of the waiting queue; when the volume to which the input/output operation waiting to be executed belongs is in the waiting queue, keeping the volume at the original position in the waiting queue; when a volume to which an input/output operation waiting for execution belongs is in an execution queue, judging the state of the volume; and when the state of the volume is an execution ending state or a destruction waiting state, deleting the volume from the execution queue and adding the volume to the tail of the waiting queue.
Specifically, in step S12, when the number of volumes to which the new input/output operation waiting to be executed belongs is several and the volume to which the new input/output operation waiting to be executed belongs is saved in the waiting queue, it is determined whether each volume is a new device or an old device.
Several, one or more.
The new device, means that the volume is not in the wait queue, nor in the execute queue. New devices, including volumes that have never been saved in the pending queue and volumes that have been deleted from the execution queue.
Old device, means the volume is in a wait queue, or in an execute queue.
In executing the storage system quality of service control method provided by the present invention, a reference count may be set for each volume. And the reference count is used for judging whether the state of the volume is a destroy waiting state or not. The initial reference count for each volume is 0.
When the volume is not in the wait queue and not in the execute queue, i.e., the volume is a new device, the volume may be directly added to the tail of the wait queue and the reference count for the volume may be incremented by one.
When the volume is in the waiting queue or the execution queue, that is, the volume is an old device, the state of the volume needs to be further judged, and corresponding processing is performed according to the state of the volume.
When the volume is in the waiting queue, that is, the state of the volume is in the waiting processing state, it is described that there are input/output operations which are not executed and belong to the volume and are waiting to be executed, the volume is kept at the original position in the waiting queue, and the reference count of the volume is not changed. If the volume is assigned a thread in a subsequent step, the volume needs to perform input/output operations a number of times that is not completed plus a new number of times.
When the volume is in the execution queue, the state of the volume is a processing state, an execution ending state or a destroy waiting state.
When the state of the volume is the processing state, the volume is processed and input/output operation waiting for execution is described, the volume is kept at the original position in the execution queue, and the reference count of the volume is not changed. Because the volume is still in the thread, the number of times the volume needs to perform the input/output operation is changed to the number of times the volume needs to perform currently plus a new number of times.
When the state of the volume is the execution end state, the volume is deleted from the execution queue and added to the queue tail of the waiting queue, and the reference count of the volume is not changed. If the volume is assigned a thread in a subsequent step, the volume needs to perform input/output operations a number of times that is not completed plus a new number of times.
When the state of the volume is the state of waiting for destruction, the I/O operation which is described to be executed and belongs to the volume before is executed, the I/O operation is added to the tail of the queue in the waiting queue, and the reference count of the volume is increased by one.
After setting a reference count for each volume, step S3 further includes: when the volume state is an execution end state after the thread becomes an idle thread, the reference count of the volume is not changed; when the state of the volume is a wait for destroy state, the reference count of the volume is decremented by one.
Through the above process, the reference count of the volume waiting for the destruction state is 0. Therefore, after step S12, it can be determined whether the state of the volume is the destroy-waiting state by whether the reference count is 0.
Based on the above embodiment, step S12 further includes: when the waiting queue is not empty, the process proceeds to step S13.
Specifically, in step S11, when there is a thread in the thread pool that is free, it is determined whether the wait queue is empty, and if the wait queue is not empty, it indicates that there is a thread to which a volume to which an input/output operation to be executed belongs is not allocated, the process goes directly from step S12 to step S13, and step S13 is executed to determine the volume to be allocated.
Based on the above embodiment, step S2 is followed by: in step S3, when the volume in any thread has finished executing all the input/output operations waiting to be executed, or the number of times of executing the input/output operations reaches the preset maximum number, the occupied thread resource is released, the thread becomes an idle thread, and the process returns to step S1.
Specifically, by step S2, after an idle thread is assigned to the volume to be allocated, the volume to be allocated performs all pending input/output operations belonging to that volume.
In step S3, the volume to be allocated performs all the pending input/output operations belonging to the volume, including two results: and when the number of times of executing the input/output operation reaches the preset maximum number, the volume executes the input/output operation forcibly, and does not execute all the input/output operation waiting to be executed.
In any case, the callback function is executed to release the thread resource occupied by the volume and make the thread idle.
When the thread becomes an idle thread, the idle thread is again present in the thread pool, and the process returns to step S1 until all the input/output operations waiting to be executed are processed.
The embodiment of the invention releases the thread resource occupied by the volume when all the input/output operations waiting for execution are executed or the number of times of executing the input/output operations reaches the preset maximum number of times, so that the thread becomes an idle thread, and the input/output task can be circularly processed until all the input/output operations belonging to each volume are executed.
The storage system service quality control method provided by the invention is described by an example.
The distributed storage system has two volumes: volume 1 and volume 2. The number of threads in the thread pool is 2. The preset maximum number of times of performing the input/output operation per volume is 50.
The input/output operation waiting for execution belonging to the volume 1 is 40 times; the input/output operations waiting to be performed belonging to volume 2 are 60 times.
Initially, 2 threads are all idle threads, and the wait queue is empty. Volume 1 and volume 2 are both saved to the wait queue and the reference counts are both 1, and are both moved to the execute queue, assigned to thread 1 and thread 2, respectively.
In the process of executing the input/output operation by the volume 1 and the volume 2, the input/output operation waiting to be executed belonging to the volume 2 is newly generated 30 times.
Volume 1 has performed 40I/O operations, the state is a wait for destroy state, and the reference count is decremented by 1, thread 1 becomes an idle thread.
The thread 1 becomes an idle thread, and stores the volume 2 to which the input/output operation waiting for execution newly occurs in the waiting queue.
When the volume 2 to which the input/output operation waiting for execution is newly generated is saved in the waiting queue, the number of times that the volume 2 executes the input/output operation has already reached 50 times, the state is the execution end state, and the thread 2 becomes an idle thread. Therefore, when the volume 2 to which the input/output operation waiting for execution newly occurs is saved in the waiting queue, since the volume 2 is the old device and the state is the execution end state, the volume 2 is deleted from the execution queue, is saved to the tail of the waiting queue, the reference count is unchanged, and is determined as the volume to be determined.
At this time, since the reference count of volume 1 is 0, the state is a destroy waiting state, volume 1 in the execution queue is deleted, and the resources occupied by volume 1 are released.
Volume 2 is determined to be the volume to be allocated, and thread 1 is allocated to volume 2. After the volume 2 has performed 60-50+ 10-40 i/o operations, the state is the wait for destroy state, the reference count is decremented by 1, and the thread 1 becomes an idle thread again.
At this time, the thread pool and the wait queue are both empty, and there is no volume to which the input/output operation waiting to be executed belongs, and since the reference count of volume 1 is 0, the state is a wait-to-destroy state, volume 2 in the execution queue is deleted, and the resource occupied by volume 2 is released.
To this end, the entire process of the storage system quality of service control method ends.
Fig. 2 is a functional block diagram of a storage system qos control apparatus according to an embodiment of the present invention. Based on the above embodiment, as shown in fig. 2, an apparatus for controlling quality of service of a storage system includes: the device determining module 21 is configured to determine a volume to be allocated according to a frequency of executing input/output operations of each volume in the waiting queue when an idle thread exists in the thread pool and the waiting queue is not empty, and add the volume to be allocated to the execution queue; a thread allocation module 22, configured to allocate an idle thread to the volume to be allocated, so that the volume to be allocated performs all the input/output operations that belong to the volume and are waiting to be performed.
Specifically, the device determination module 21 is electrically connected to the thread assignment module 22.
After the device determining module 21 determines the volume to be allocated, the thread allocating module 22 allocates an idle thread to the volume to be allocated determined by the device determining module 21.
The storage system service quality control device provided by the invention is used for executing the storage system service quality control method provided by the invention. The specific method and flow for each module included in the storage system qos control apparatus to implement the corresponding function are detailed in the above embodiment of the storage system qos control method, and are not described herein again.
The embodiment of the invention determines the volume to be allocated according to the frequency of executing the input/output operation by each volume in the waiting queue, and allocates the idle thread to the volume to be allocated, so that the volume to be allocated executes all the input/output operations to be executed belonging to the volume, and the fairness and the continuity can be considered when the quality of service of the storage system is controlled under the condition that a plurality of volumes have the input/output operations to be executed, and the quality of service of the storage system is improved. Furthermore, the storage system service quality control can be performed more conveniently and quickly without setting the IOPS, MBPS, weight and the like of each volume independently.
Based on the above embodiment, the device determination module includes: the judging submodule judges whether the waiting queue is empty or not when an idle thread exists in the thread pool; the queue submodule is used for storing the volume to which the input/output operation to be executed belongs into the waiting queue when the waiting queue is empty; the selection submodule determines the volume to be allocated according to the frequency of executing input/output operation of each volume in the waiting queue, deletes the volume to be allocated from the waiting queue and adds the volume to be allocated to the execution queue; and the recovery submodule is used for deleting the volume in the execution queue in the state of waiting for destruction and releasing the memory space occupied by the volume in the execution queue in the state of waiting for destruction.
Specifically, the judgment submodule is electrically connected with the queue submodule and the selection submodule respectively; the queue submodule is electrically connected with the selection submodule and the recovery submodule respectively.
The storage system service quality control device provided by the invention is used for executing the storage system service quality control method provided by the invention. The specific method and flow for each sub-module included in the device determination module of the storage system service quality control apparatus to implement the corresponding function are detailed in the above embodiment of the storage system service quality control method, and are not described herein again.
According to the embodiment of the invention, the volume to which the newly generated input/output operation to be executed belongs is stored in the waiting queue when the thread pool has an idle thread and the waiting queue is empty, so that the waiting queue is not empty, the input/output task can be circularly processed until all the input/output operations belonging to each volume are executed, and all the input/output operations to be executed can be executed more quickly under the condition of multi-thread concurrence. Furthermore, the resources occupied by the volumes waiting for destruction are recycled after the volumes to which the input/output operations waiting for execution belong are saved in the waiting queue, so that the system resources are more effectively utilized, and the service quality of the storage system can be improved.
Finally, the above-mentioned embodiments of the present invention are merely preferred embodiments, and are not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (7)
1. A storage system service quality control method is characterized by comprising the following steps:
s1, when an idle thread exists in the thread pool and the waiting queue is not empty, determining the volume to be allocated according to the frequency of executing input/output operation of each volume in the waiting queue;
s2, allocating an idle thread to the volume to be allocated, so that the volume to be allocated executes all input/output operations which belong to the volume and are waiting to be executed;
the S1 includes:
s11, when an idle thread exists in the thread pool, judging whether the waiting queue is empty;
s12, when the waiting queue is empty, storing the volume to which the input/output operation waiting to be executed belongs into the waiting queue;
s13, determining the volume to be distributed according to the frequency of executing input/output operation of each volume in the waiting queue, deleting the volume to be distributed from the waiting queue, and adding the volume to be distributed into the executing queue;
the state of the volume includes: a waiting processing state, a processing state, an execution ending state and a waiting destroying state;
when the volume is in the waiting queue, the state of the volume is a waiting processing state;
when the volume does not finish all input/output operations waiting to be executed and the number of times of executing the input/output operations does not reach the preset maximum number, the state of the volume is a processing state;
when the volume does not finish all input/output operations waiting to be executed and the number of times of executing the input/output operations reaches the preset maximum number, the state of the volume is an execution ending state;
when the volume finishes executing all input/output operations waiting to be executed, the state of the volume is a state waiting to be destroyed.
2. The storage system service quality control method according to claim 1, wherein the step S12 is followed by further comprising:
and deleting the volume in the execution queue in the state of waiting for destruction, and releasing the resources occupied by the volume.
3. The storage system qos control method according to any one of claims 1 to 2, wherein the specific step of determining the volume to be allocated according to the frequency of executing the i/o operation on each volume in the wait queue includes:
sequentially calculating the average value of the input/output operation times per second of each volume of the waiting queue according to the sequence from the head of the queue to the tail of the queue until the average value is smaller than a preset threshold value, and determining the volume of which the average value is smaller than the preset threshold value as the volume to be distributed;
wherein the average of the number of times of performing the input/output operation per second is an average of the number of times of performing the input/output operation per second of the volume from when the input/output request starts to be processed to when the average is calculated.
4. The storage system service quality control method according to claim 1, wherein the S12 specifically includes:
when the volume to which the input/output operation waiting to be executed belongs is not in the waiting queue and is not in the execution queue, adding the volume to the tail of the waiting queue;
when the volume to which the input/output operation waiting to be executed belongs is in the waiting queue, keeping the volume at the original position in the waiting queue;
when a volume to which an input/output operation waiting for execution belongs is in an execution queue, judging the state of the volume; and when the state of the volume is an execution ending state or a destruction waiting state, deleting the volume from the execution queue and adding the volume to the tail of the waiting queue.
5. The storage system quality of service control method according to claim 1 or 4, wherein the S12 further includes:
when the wait queue is not empty, the process proceeds to S13.
6. The storage system service quality control method according to claim 1, wherein the step S2 is followed by further comprising:
s3, when the volume in any thread finishes executing all the I/O operations waiting to be executed, or the number of executing I/O operations reaches the preset maximum number, releasing the occupied thread resource, making the thread become an idle thread, and returning to execute S1.
7. A storage system quality of service control apparatus, comprising:
the device determining module is used for determining the volume to be allocated according to the frequency of executing input/output operations of each volume in the waiting queue when an idle thread exists in the thread pool and the waiting queue is not empty, and adding the volume to be allocated to the execution queue;
the thread allocation module is used for allocating an idle thread to the volume to be allocated so that the volume to be allocated executes all input/output operations which belong to the volume and are waiting to be executed;
the device determination module includes:
the judging submodule judges whether the waiting queue is empty or not when an idle thread exists in the thread pool;
the queue submodule is used for storing the volume to which the input/output operation to be executed belongs into the waiting queue when the waiting queue is empty;
the selection submodule determines a volume to be allocated according to the frequency of executing input/output operation of each volume in the waiting queue, deletes the volume to be allocated from the waiting queue and adds the volume to be allocated to the execution queue;
and the recovery submodule is used for deleting the volume in the execution queue in the state of waiting for destruction and releasing the resources occupied by the volume.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810089640.2A CN108459825B (en) | 2018-01-30 | 2018-01-30 | Storage system service quality control method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810089640.2A CN108459825B (en) | 2018-01-30 | 2018-01-30 | Storage system service quality control method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108459825A CN108459825A (en) | 2018-08-28 |
| CN108459825B true CN108459825B (en) | 2020-12-29 |
Family
ID=63239361
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810089640.2A Active CN108459825B (en) | 2018-01-30 | 2018-01-30 | Storage system service quality control method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108459825B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109491610A (en) * | 2018-11-07 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of memory bandwidth analysis method and device based on hard disk configuration |
| CN109445714A (en) * | 2018-11-09 | 2019-03-08 | 浪潮电子信息产业股份有限公司 | A kind of recycle bin method for cleaning and device |
| CN113625954B (en) * | 2021-06-30 | 2024-03-15 | 济南浪潮数据技术有限公司 | Method, device and equipment for inquiring volume information and readable storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7711872B2 (en) * | 2007-10-02 | 2010-05-04 | Hitachi, Ltd. | Storage apparatus, process controller, and storage system |
| CN101739293A (en) * | 2009-12-24 | 2010-06-16 | 航天恒星科技有限公司 | Method for scheduling satellite data product production tasks in parallel based on multithread |
| CN103049574A (en) * | 2013-01-04 | 2013-04-17 | 中国科学院高能物理研究所 | Key value system and key value method for implementation of dynamic duplicates of documents |
| CN103577257A (en) * | 2012-08-03 | 2014-02-12 | 杭州勒卡斯广告策划有限公司 | REST (representational state transfer) service method, device and system |
| CN106155803A (en) * | 2015-04-07 | 2016-11-23 | 北大方正集团有限公司 | A kind of thread pool sharing method based on semaphore and system |
| CN106294472A (en) * | 2015-06-03 | 2017-01-04 | 中国移动通信集团广东有限公司 | The querying method of a kind of Hadoop data base HBase and device |
| CN106293523A (en) * | 2016-08-04 | 2017-01-04 | 浪潮(北京)电子信息产业有限公司 | A kind of I/O Request response method to non-volatile memories and device |
| CN107133100A (en) * | 2017-04-26 | 2017-09-05 | 新华三技术有限公司 | Storage system service quality Q oS control methods and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8055624B2 (en) * | 2004-12-09 | 2011-11-08 | International Business Machines Corporation | On-site reclamation of off-site copy storage volumes using multiple, parallel processes |
-
2018
- 2018-01-30 CN CN201810089640.2A patent/CN108459825B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7711872B2 (en) * | 2007-10-02 | 2010-05-04 | Hitachi, Ltd. | Storage apparatus, process controller, and storage system |
| CN101739293A (en) * | 2009-12-24 | 2010-06-16 | 航天恒星科技有限公司 | Method for scheduling satellite data product production tasks in parallel based on multithread |
| CN103577257A (en) * | 2012-08-03 | 2014-02-12 | 杭州勒卡斯广告策划有限公司 | REST (representational state transfer) service method, device and system |
| CN103049574A (en) * | 2013-01-04 | 2013-04-17 | 中国科学院高能物理研究所 | Key value system and key value method for implementation of dynamic duplicates of documents |
| CN106155803A (en) * | 2015-04-07 | 2016-11-23 | 北大方正集团有限公司 | A kind of thread pool sharing method based on semaphore and system |
| CN106294472A (en) * | 2015-06-03 | 2017-01-04 | 中国移动通信集团广东有限公司 | The querying method of a kind of Hadoop data base HBase and device |
| CN106293523A (en) * | 2016-08-04 | 2017-01-04 | 浪潮(北京)电子信息产业有限公司 | A kind of I/O Request response method to non-volatile memories and device |
| CN107133100A (en) * | 2017-04-26 | 2017-09-05 | 新华三技术有限公司 | Storage system service quality Q oS control methods and device |
Non-Patent Citations (4)
| Title |
|---|
| Improving OpenStack Swift interaction with the I/O Stack to Enable Software Defined Storage;R. Nou et al.;《2017 IEEE 7th International Symposium on Cloud and Service Computing (SC2), Kanazawa》;20171125;全文 * |
| iostat 磁盘io分析工具;九卷;《https://www.cnblogs.com/jiujuan/p/9027265.html》;20160512;全文 * |
| 一种基于请求大小的固态盘I/O调度算法;吴素贞 等;《计算机工程》;20140131;第40卷(第1期);全文 * |
| 基于存储虚拟化系统的QoS控制系统的设计与实现;刘云剑;《中国优秀硕士学位论文全文数据库 信息科技辑》;20170315;第I139-308页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108459825A (en) | 2018-08-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10387202B2 (en) | Quality of service implementation in a networked storage system with hierarchical schedulers | |
| US10185592B2 (en) | Network storage device using dynamic weights based on resource utilization | |
| US9930133B2 (en) | System and method for managing application performance | |
| US20170206107A1 (en) | Systems And Methods For Provisioning Of Storage For Virtualized Applications | |
| US9619287B2 (en) | Methods and system for swapping memory in a virtual machine environment | |
| CN108459825B (en) | Storage system service quality control method and device | |
| US8458712B2 (en) | System and method for multi-level preemption scheduling in high performance processing | |
| WO2016078178A1 (en) | Virtual cpu scheduling method | |
| JP5789894B2 (en) | Buffer manager and memory management method | |
| JP2003256221A (en) | Parallel process executing method and multi-processor type computer | |
| JP3876100B2 (en) | A system that manages access to devices | |
| US20130061233A1 (en) | Efficient method for the scheduling of work loads in a multi-core computing environment | |
| US20190332328A1 (en) | Storage Controller and IO Request Processing Method | |
| US20170123975A1 (en) | Centralized distributed systems and methods for managing operations | |
| KR20130033020A (en) | Apparatus and method for partition scheduling for manycore system | |
| WO2017141413A1 (en) | Computer, communication driver, and communication control method | |
| US10534712B1 (en) | Service level agreement based management of a pre-cache module | |
| CN112506634B (en) | Fairness operation scheduling method based on reservation mechanism | |
| US20190286582A1 (en) | Method for processing client requests in a cluster system, a method and an apparatus for processing i/o according to the client requests | |
| JP3664021B2 (en) | Resource allocation method by service level | |
| CN113010309B (en) | Cluster resource scheduling method, device, storage medium, equipment and program product | |
| EP3293625B1 (en) | Method and device for accessing file, and storage system | |
| CN110908790B (en) | Memory storage device and operation method thereof | |
| JPH09244931A (en) | Real-time file system | |
| CN114548389A (en) | Management method of computing unit in heterogeneous computing and corresponding processor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220328 Address after: No. 407, floor 4, No. 9, No. 9, shangdijiu street, Haidian District, Beijing 100085 Patentee after: Shenzhou Yunke (Beijing) Technology Co.,Ltd. Address before: 518131 F3, 11th floor, No. 8 Kefa Road, Yuehai Street, Nanshan District, Shenzhen City, Guangdong Province Patentee before: Shenzhen science and Technology Co.,Ltd. digital cloud data |