CN112905345B - Task allocation method, distributed storage system and server - Google Patents
Task allocation method, distributed storage system and server Download PDFInfo
- Publication number
- CN112905345B CN112905345B CN202110204142.XA CN202110204142A CN112905345B CN 112905345 B CN112905345 B CN 112905345B CN 202110204142 A CN202110204142 A CN 202110204142A CN 112905345 B CN112905345 B CN 112905345B
- Authority
- CN
- China
- Prior art keywords
- disk
- node devices
- server
- state data
- input
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000004891 communication Methods 0.000 claims description 9
- 238000001514 detection method Methods 0.000 claims description 6
- 239000000758 substrate Substances 0.000 claims 3
- 238000013500 data storage Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000005299 abrasion Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
Abstract
The application relates to the technical field of data storage, and discloses a task allocation method, a distributed storage system and a server. The method comprises the following steps: the server acquires state data of each disk of a plurality of node devices, wherein the state data comprises disk input/output randomness information; and the server performs task allocation in a plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance. By the method, the storage resource utilization rate of the magnetic disk in the plurality of node devices can be improved, and further the storage service capacity of the distributed storage system formed by the server and the plurality of node devices is improved.
Description
Technical Field
The present disclosure relates to the field of data storage technologies, and in particular, to a task allocation method, a distributed storage system, a server, and a computer readable storage medium.
Background
With the development of novel technologies such as cloud computing and big data, a large amount of data is generated, the requirement on storage is also higher and higher, and the performance, stability and reliability of storage are focused on the distributed storage system. In addition, in order to adapt to data processing under a large data volume, the requirements on the performance of stored applications are higher and higher, and the complexity of the applications is higher and higher.
In the whole distributed storage system, the performances of the magnetic disks of the node devices are different, and the difference of the performances of the magnetic disks of the node devices becomes larger along with the increase of the service time and the different wear degrees, so that the problem of overload of the magnetic disks occurs and the storage service capacity of the distributed storage system is affected.
Disclosure of Invention
The technical problem to be solved mainly by the application is to provide a task allocation method, a distributed storage system, a server and a computer readable storage medium, which can improve the utilization rate of storage resources of magnetic disks in a plurality of node devices, and further improve the storage service capacity of the distributed storage system formed by the server and the plurality of node devices.
The technical scheme adopted by the application is to provide a task allocation method, which comprises the following steps: the server acquires state data of each disk of a plurality of node devices, wherein the state data comprises disk input/output randomness information; and the server performs task allocation in a plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
The task allocation of the server in the plurality of node devices according to the state data comprises the following steps: the server obtains the input and output randomness information of the disk from the state data; the server compares the disk input and output randomness information with the randomness threshold value, and performs task allocation in a plurality of node devices according to the comparison result.
The task allocation in the plurality of node devices according to the comparison result comprises the following steps: obtaining the operation saturation of the magnetic disk according to the comparison result; and performing task allocation in a plurality of node devices according to the operation saturation of the disk.
Wherein, the status data also comprises read-write delay information; obtaining the operation saturation of the magnetic disk according to the comparison result comprises the following steps: and obtaining the operation saturation according to the load information, the read-write delay information and the comparison result of the plurality of node devices.
The task allocation in the plurality of node devices according to the disk operation saturation comprises the following steps: and the tasks of the node devices with the disk operation saturation higher than the operation saturation threshold value in the plurality of node devices are distributed to the node devices with the disk operation saturation lower than the operation saturation threshold value.
The task allocation of the server in the plurality of node devices according to the state data comprises the following steps: and a task allocation module in the server allocates tasks in a plurality of node devices according to the state data.
Another technical solution adopted in the present application is to provide a distributed storage system, which includes: a plurality of node devices; the server is in communication connection with the plurality of node devices and is used for acquiring state data of each disk of the plurality of node devices, wherein the state data comprises disk input and output randomness information; and task allocation is carried out in a plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
The plurality of node devices comprise disk state detection modules, and are used for collecting state data of respective disks of the plurality of node devices.
Another technical scheme adopted by the application is to provide a server, which comprises a processor, a memory connected with the processor and a communication circuit; the communication circuit is used for communicating with the node equipment, the memory is used for storing program data, and the processor is used for executing the program data so as to realize the task allocation method provided by the technical scheme.
Another technical solution adopted in the present application is to provide a computer readable storage medium, where the computer readable storage medium is used to store program data, and the program data is used to implement the task allocation method provided in the above technical solution when the program data is executed by a processor.
The beneficial effects of this application are: different from the condition of the prior art, the server utilizes the state data of the disks in the plurality of node devices to allocate the tasks from the disks with low disk performance to the disks with high disk performance, so that the dynamic allocation of the tasks is realized, the utilization rate of storage resources of the disks in the plurality of node devices can be improved, and further the storage service capacity of a distributed storage system formed by the server and the plurality of node devices is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a schematic diagram illustrating the structure of one embodiment of a distributed storage system provided herein;
FIG. 2 is a flow chart of an embodiment of a task orchestration method provided herein;
FIG. 3 is a flow chart illustrating another embodiment of a task orchestration method provided herein;
FIG. 4 is a schematic illustration of a specific flow chart of step 32 of FIG. 3 provided herein;
FIG. 5 is a schematic diagram of the specific flow of step 322 in FIG. 4 provided herein;
FIG. 6 is a flow chart of another embodiment of a task orchestration method provided herein;
FIG. 7 is a schematic diagram of another embodiment of a distributed storage system provided herein;
FIG. 8 is a schematic diagram illustrating interaction between a node device and a server in a distributed storage system provided herein;
FIG. 9 is a schematic diagram of an embodiment of a server provided herein;
fig. 10 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the specific embodiments described herein are for purposes of illustration only and are not limiting. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present application are shown in the drawings. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
Referring to fig. 1, fig. 1 is a schematic structural diagram of an embodiment of a distributed storage system provided in the present application. The distributed storage system 10 includes a server 11 and a number of node devices 12. The distributed storage system 10 is a distributed storage system that disperses data across disks of a number of node devices 12. The distributed storage system 10 adopts an expandable system structure, utilizes a plurality of node devices 12 to share storage load, utilizes a server 11 to position storage information, improves the reliability, availability and access efficiency of the system, and is easy to expand.
Referring to fig. 2, fig. 2 is a flow chart of an embodiment of a task allocation method provided in the present application. The method comprises the following steps:
step 21: the server acquires state data of respective disks of a plurality of node devices, wherein the state data comprises input and output randomness information of the disks.
In this embodiment, the server is communicatively connected to a plurality of node devices, and the server may assign a corresponding task to each node device. The disk in the corresponding node device will perform read-write operation to respond to the task. Thus, when a task is running, a corresponding input-output request is generated. When multiple tasks are running simultaneously, a large number of input-output requests are generated. The node device can calculate corresponding disk input-output randomness information according to the input-output requests.
Specifically, the input-output requests include random input-output requests and sequential input-output requests. When calculating the random degree information of the input and output of the magnetic disk, calculating the total number of the random input and output requests and the sequential input and output requests, then calculating the duty ratio of the random input and output requests in the total number, and taking the duty ratio as the random degree information of the input and output of the magnetic disk.
It can be understood that different disk tasks correspond to a disk input/output random degree information, each task is provided with a corresponding random degree threshold, and when the disk input/output random degree information exceeds the random degree threshold, the disk performance of the node equipment is determined to be low; and when the random degree information of the disk input and output does not exceed the random degree threshold value, determining that the disk performance of the node equipment is high.
For example, the operating system of the node device is a Linux operating system. The disk input output randomness information may be determined by customizing the Linux kernel based on the average request queue size or the average read and write latency.
The server can acquire the state data of the respective disks of the node devices at regular time.
The magnetic disks in the plurality of node devices can be mechanical hard disks or solid state hard disks.
Step 22: and the server performs task allocation in a plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
In some embodiments, the server may determine disk performance of the corresponding node device based on the disk input-output randomness information. And allocating tasks belonging to node equipment with low disk performance to node equipment with high disk performance so as to balance the load of disk use in a plurality of node equipment.
Specifically, according to the fact that the disk of each node device is provided with a randomness threshold, after the server acquires the state data of any node device, acquiring disk input and output randomness information in the state data, comparing the disk input and output randomness information with the randomness threshold, and determining that the disk performance of the node device is low when the disk input and output randomness information exceeds the randomness threshold; and when the randomness information of the disk input and output is smaller than or equal to the randomness threshold, determining that the disk performance of the node equipment is high.
In an application scenario, the distributed storage system includes a server and node device 1, a node device 2, a node device 3, and a node device N, where N is a positive integer and greater than 3. The server acquires state data of respective disks of the node equipment 1, the node equipment 2, the node equipment 3 and the node equipment N, wherein the state data comprises disk input and output randomness information; the server determines that the disk performance of the node device 1 is high, the disk performance of the node device 2 is low, the disk performance of the node device 3 is high and the disk performance of the node device N is low according to the disk input/output randomness information. The task allocation module of the server allocates the task of the node equipment 2 to the node equipment 1 in combination with the disk performance of each node equipment, and allocates the task of the node equipment N to the node equipment 3 so as to balance the loads of the disk use in the node equipment 1, the node equipment 2, the node equipment 3 and the node equipment N.
In this embodiment, the server utilizes state data of the disks in the plurality of node devices to allocate tasks from the disk with low disk performance to the disk with high disk performance, so as to implement dynamic allocation of tasks, and improve the utilization rate of storage resources of the disk in the plurality of node devices, thereby improving the storage service capacity of the distributed storage system formed by the server and the plurality of node devices.
Referring to fig. 3, fig. 3 is a flow chart of another embodiment of the task allocation method provided in the present application. The method comprises the following steps:
step 31: the server acquires state data of respective disks of a plurality of node devices.
Step 32: the server obtains the disk input-output randomness information from the state data.
Steps 31 to 32 have the same or similar technical solutions as the above embodiments, and are not described here again.
Step 33: the server compares the disk input and output randomness information with the randomness threshold value, and performs task allocation in a plurality of node devices according to the comparison result.
Specifically, referring to fig. 4, task allocation in a plurality of node devices according to the comparison result may be the following steps:
step 331: and obtaining the operation saturation of the magnetic disk according to the comparison result.
In some embodiments, the status data may also include the percentage of time the disk is active (bandwidth usage of the drive), the amount of data transferred (read or write) to the drive in KB per second, the total amount of data read, the total amount of data written, and the number of transfers per second to the disk, where a transfer is one input/output request to the disk, and multiple logical requests may be combined into one single input/output request to the disk. And comprehensively calculating the operation saturation of the magnetic disk according to the data and the comparison result.
In some embodiments, when the server obtains the state data of the respective disks of the plurality of node devices, the server also obtains load information and read-write delay information of the plurality of node devices. Then, when step 331 is executed, the operational saturation may be obtained by combining the load information, the read-write delay information and the comparison result of the plurality of node devices. Specifically, different weights can be set for the load information, the read-write delay information and the comparison result, and if the comparison result is that the input-output randomness information of the disk is smaller than the randomness threshold, the weight corresponding to the comparison result is set as the basic weight. If the comparison result is that the random degree information input and output by the magnetic disk is larger than the random degree threshold value, setting the weight corresponding to the comparison result to be higher than the basic weight. If the basic weight is 0.4, changing the basic weight into 0.6 when the comparison result shows that the random degree information of the input and output of the magnetic disk is larger than the random degree threshold value, and correspondingly reducing the weights of the other parameters.
Step 332: and performing task allocation in a plurality of node devices according to the operation saturation of the disk.
In some embodiments, determining that disk performance of the node device is low when the disk operational saturation is above a disk operational saturation threshold; and when the disk operation saturation is lower than the disk operation saturation threshold, determining that the disk performance of the node equipment is high. The task of the node device with the disk operation saturation higher than the disk operation saturation threshold can be allocated to the node device with the disk operation saturation lower than the disk operation saturation threshold.
Specifically, the disk operation saturation of a plurality of node devices can be ordered in the order from big to small, and the task of the node device with the top rank in the order is allocated to the node device with the top rank. If the number of the node devices is 100, after the ordering is finished according to the order from the high to the low according to the disk operation saturation, the tasks of the node devices with the top 10 bits are allocated to the node devices with the top 10 bits.
In other embodiments, referring to fig. 5, the specific steps of step 332 are as follows:
step 3321: and obtaining the average disk operation saturation by utilizing the disk operation saturation of the plurality of node devices.
Step 3322: and the tasks of the partial plurality of node devices with the disk operation saturation higher than the average disk operation saturation are distributed to the partial plurality of node devices with the disk operation saturation lower than the average disk operation saturation.
Specifically, the plurality of node devices with the disk operation saturation higher than the average disk operation saturation and the plurality of node devices with the disk operation saturation lower than the average disk operation saturation are respectively ordered in the order from the big to the small. For example, the number of node devices is 100, the number of node devices with disk operation saturation higher than the average disk operation saturation is 30, and the number of node devices with disk operation saturation lower than the average disk operation saturation is 70. The tasks of 30 node devices above the average disk running saturation may be formulated for the last 30 bit node device of the 70 node devices below the average disk running saturation.
In other embodiments, the magnetic disk in the node device is a mechanical hard disk, and many mechanical components of the mechanical hard disk mainly comprise a disk, a magnetic head, a disk rotating shaft, a control motor, a magnetic head controller, a data converter, an interface, a cache and the like. The performance parameters are complex, so that the overall performance difference of different magnetic discs is large, and the overall performance difference of the magnetic discs becomes larger as the service time is prolonged and the abrasion degree is different. According to the method of the embodiment, the server can allocate corresponding tasks to the mechanical hard disks with different performances of the plurality of node devices so as to avoid the phenomenon of oversaturation of the mechanical hard disks, improve the utilization rate of storage resources of the mechanical hard disks in the plurality of node devices, and further improve the storage service capacity of the distributed storage system formed by the server and the plurality of node devices.
Referring to fig. 6, fig. 6 is a flowchart of another embodiment of a task allocation method provided in the present application.
Step 61: the server acquires state data of respective disks of a plurality of node devices.
Step 62: the server acquires the disk input/output randomness information of each task in the state data.
In this embodiment, the server may obtain the disk input/output randomness information of each task in the node device. It will be appreciated that after each task is determined, its corresponding disk input/output request is determined, and then each task may correspond to a disk input/output randomness information.
Step 63: the server compares the disk input and output randomness information of each task with the randomness threshold corresponding to each task, and performs task allocation in a plurality of node devices according to the comparison result.
The randomness threshold corresponding to each task is obtained by executing the task according to a normal disk in advance.
When the random degree information of the disk input and output of any task is larger than the corresponding random degree threshold value, the random degree information of the disk input and output of the task is used as the random degree information of the disk input and output of the node equipment. And obtaining the disk operation saturation of the node equipment according to the disk input/output randomness information. And performing task allocation in a plurality of node devices according to the operation saturation of the disk.
In an application scenario, tasks a, B, and C exist in a disk of the node device. The server acquires the disk input/output randomness information of the corresponding task A, the disk input/output randomness information of the task B and the disk input/output randomness information of the task C in the disk of the node equipment. And comparing the disk input/output randomness information of the task A with the randomness threshold of the task A, comparing the disk input/output randomness information of the task B with the randomness threshold of the task B, and comparing the disk input/output randomness information of the task C with the randomness threshold of the task C. If any comparison result is larger than the corresponding randomness threshold, determining that the performance of the magnetic disk is low. And the operation saturation of the disk can be calculated according to the load information and the read-write delay information of the node equipment.
Specifically, different weights can be set for the load information, the read-write delay information and the comparison result, and if the comparison result is that the input-output randomness information of the disk is smaller than the randomness threshold, the weight corresponding to the comparison result is set as the basic weight. If the comparison result is that the random degree information input and output by the magnetic disk is larger than the random degree threshold value, setting the weight corresponding to the comparison result to be higher than the basic weight. If the basic weight is 0.3, changing the basic weight into 0.5 when the comparison result shows that the input and output randomness information of the disk is larger than the randomness threshold, and correspondingly reducing the weights of the other parameters.
In other embodiments, a task orchestration module is provided in the server. And the task allocation module allocates tasks in a plurality of node devices according to the state data. Specifically, the operation saturation of the disk is obtained according to the state data, and then task allocation is performed according to the method of the embodiment by combining a plurality of operation saturation.
In this embodiment, the server utilizes state data of the disks in the plurality of node devices to allocate tasks from the disk with low disk performance to the disk with high disk performance, so as to implement dynamic allocation of tasks, and improve the utilization rate of storage resources of the disk in the plurality of node devices, thereby improving the storage service capacity of the distributed storage system formed by the server and the plurality of node devices.
Referring to fig. 7, a distributed storage system 70 includes a number of node devices 71 and a server 72. The server 72 is in communication connection with the plurality of node devices 71, and is configured to obtain status data of respective disks of the plurality of node devices, where the status data includes disk input/output randomness information; and task allocation is carried out in a plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
The plurality of node devices 71 each include a disk state detection module for collecting state data of respective disks of the plurality of node devices 71.
In an application scenario, the following is described with reference to fig. 8: as shown in fig. 8, the server 72 includes a status data acquisition module 721 and a task orchestration module 722. The node device 71 includes a disk state detection module 711 and a task management module 712.
The task management module 712 is configured to receive a task issued by the task allocation module 722, so that the node device 71 performs the task. The disk state detection module 711 is configured to collect state data of a disk, such as disk input/output randomness information, disk read/write delay information, and disk usage rate.
The status data acquisition module 721 is for acquiring status data in the node apparatus 71. Such as may be periodically acquired or periodically uploaded by the disk state detection module 711 in the node device 71.
In other embodiments, the status data acquiring module 721 may acquire status data of the plurality of node devices 71, and may calculate a respective disk operating saturation of the plurality of node devices 71 according to the status data. When the task allocation module 722 allocates tasks, task allocation is performed according to the disk operation saturation of the plurality of node devices 71, and the tasks belonging to the node devices 71 with high disk operation saturation are allocated to the node devices 71 with low disk operation saturation so as to balance the loads of the disks in the plurality of node devices 71. Specifically, reference may be made to the method in the above embodiment, and details are not described here.
In other embodiments, the server 72 is also configured to implement the method of any of the embodiments described above.
In this embodiment, the server 72 utilizes the state data of the disks in the plurality of node devices 71 to implement task dynamic allocation, so that the load of the disks in the plurality of node devices 71 can be balanced, the utilization rate of storage resources of the disks in the plurality of node devices 71 can be improved, and further the storage service capacity of the distributed storage system 70 formed by the server 72 and the plurality of node devices 71 can be improved.
Referring to fig. 9, fig. 9 is a schematic structural diagram of an embodiment of a server provided in the present application. The server 90 includes a processor 91, a memory 92 connected to the processor 91, and a communication circuit 93; wherein the communication circuit 93 is for communicating with the node device, the memory 92 is for storing program data, and the processor is for executing the program data to implement the following method:
the server acquires state data of each disk of a plurality of node devices, wherein the state data comprises disk input/output randomness information; and the server performs task allocation in a plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
It will be appreciated that the processor 91 in this embodiment may also implement any of the methods in the above embodiments, and will not be described herein.
By implementing the method, the server 90 in this embodiment utilizes the state data of the disks in the plurality of node devices to allocate the tasks from the disk with low disk performance to the disk with high disk performance, so as to implement dynamic allocation of the tasks, and improve the storage resource utilization rate of the disk in the plurality of node devices, thereby improving the storage service capacity of the distributed storage system formed by the server and the plurality of node devices.
Referring to fig. 10, fig. 10 is a schematic structural diagram of an embodiment of a computer readable storage medium provided in the present application. The computer readable storage medium 100 is for storing program data 101, which program data 101, when executed by a processor, is for implementing the method of:
the server acquires state data of each disk of a plurality of node devices, wherein the state data comprises disk input/output randomness information; and the server performs task allocation in a plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
It will be appreciated that the computer readable storage medium 100 in this embodiment may also implement any of the methods of the above embodiments, and will not be described here again.
When the computer readable storage medium 100 of the embodiment is applied to the server, the method is implemented, and the server utilizes the state data of the disks in the plurality of node devices to allocate the tasks from the disk with low disk performance to the disk with high disk performance, so as to implement dynamic allocation of the tasks, and improve the utilization rate of storage resources of the disks in the plurality of node devices, thereby improving the storage service capacity of the distributed storage system formed by the server and the plurality of node devices.
In the several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses may be implemented in other manners. For example, the above-described device embodiments are merely illustrative, e.g., the division of the above modules or units is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist alone physically, or two or more units may be integrated into one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units of the other embodiments described above may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as stand alone products. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution, in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing description is only of embodiments of the present application, and is not intended to limit the scope of the patent application, and all equivalent structures or equivalent processes using the descriptions and the contents of the present application or other related technical fields are included in the scope of the patent application.
Claims (10)
1. A method of task orchestration, the method comprising:
the method comprises the steps that a server obtains state data of respective disks of a plurality of node devices, wherein the state data comprise input and output randomness information of the disks; the disk input-output randomness information is the proportion of random input-output requests in the total number, and the total number is the total number of the random input-output requests and the sequential input-output requests;
and the server performs task allocation in the plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
2. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the server performing task allocation in the plurality of node devices according to the state data comprises:
the server obtains the disk input-output randomness information from the state data;
and the server compares the disk input and output randomness information with a randomness threshold value, and performs task allocation in the plurality of node devices according to the comparison result.
3. The method of claim 2, wherein the step of determining the position of the substrate comprises,
the task allocation in the plurality of node devices according to the comparison result comprises the following steps:
obtaining the operation saturation of the magnetic disk according to the comparison result;
and performing task allocation in the plurality of node devices according to the disk operation saturation.
4. The method of claim 3, wherein the step of,
the state data also comprises read-write delay information;
the obtaining the operation saturation of the magnetic disk according to the comparison result comprises the following steps:
and obtaining the operation saturation according to the load information of the plurality of node devices, the read-write delay information and the comparison result.
5. The method of claim 3, wherein the step of,
the task allocation in the plurality of node devices according to the disk operation saturation comprises the following steps:
obtaining average disk operation saturation by using the disk operation saturation of the plurality of node devices;
and the tasks of the node devices with the disk operation saturation higher than the average disk operation saturation are distributed to the node devices with the disk operation saturation lower than the average disk operation saturation.
6. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the server performing task allocation in the plurality of node devices according to the state data comprises:
and a task allocation module in the server allocates tasks in the plurality of node devices according to the state data.
7. A distributed storage system, the distributed storage system comprising:
a plurality of node devices;
the server is in communication connection with the plurality of node devices and is used for acquiring state data of each disk of the plurality of node devices, wherein the state data comprises disk input and output randomness information; the disk input-output randomness information is the proportion of random input-output requests in the total number, and the total number is the total number of the random input-output requests and the sequential input-output requests; and performing task allocation in the plurality of node devices according to the state data so as to allocate the tasks from the disk with low disk performance to the disk with high disk performance.
8. The distributed storage system of claim 7, wherein,
the plurality of node devices comprise disk state detection modules, and are used for collecting state data of respective disks of the plurality of node devices.
9. A server, characterized in that the server comprises a processor, a memory connected with the processor and a communication circuit;
wherein the communication circuit is for communicating with a node device, the memory is for storing program data, and the processor is for executing the program data to implement the method of any of claims 1-6.
10. A computer readable storage medium for storing program data which, when executed by a processor, is adapted to carry out the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110204142.XA CN112905345B (en) | 2021-02-23 | 2021-02-23 | Task allocation method, distributed storage system and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110204142.XA CN112905345B (en) | 2021-02-23 | 2021-02-23 | Task allocation method, distributed storage system and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112905345A CN112905345A (en) | 2021-06-04 |
| CN112905345B true CN112905345B (en) | 2024-04-05 |
Family
ID=76106749
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110204142.XA Active CN112905345B (en) | 2021-02-23 | 2021-02-23 | Task allocation method, distributed storage system and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112905345B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104580439A (en) * | 2014-12-30 | 2015-04-29 | 创新科存储技术（深圳）有限公司 | Method for achieving uniform data distribution in cloud storage system |
| CN109144411A (en) * | 2018-07-24 | 2019-01-04 | 中国电子科技集团公司第三十八研究所 | Data center's hybrid magnetic disc array and its data dynamic migration strategy |
| CN109783000A (en) * | 2017-11-10 | 2019-05-21 | 成都华为技术有限公司 | A kind of data processing method and equipment |
| CN109992217A (en) * | 2019-04-11 | 2019-07-09 | 苏州浪潮智能科技有限公司 | A kind of method for controlling quality of service, device, electronic equipment and storage medium |
| CN110764706A (en) * | 2019-10-25 | 2020-02-07 | 普联技术有限公司 | Storage system, data management method, and storage medium |
| CN111290699A (en) * | 2018-12-07 | 2020-06-16 | 杭州海康威视系统技术有限公司 | Data migration method, device and system |
| CN112379825A (en) * | 2019-09-24 | 2021-02-19 | 北京城建设计发展集团股份有限公司 | Distributed data storage method and device based on data feature sub-pools |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11100420B2 (en) * | 2014-06-30 | 2021-08-24 | Amazon Technologies, Inc. | Input processing for machine learning |
| US9594839B2 (en) * | 2014-12-22 | 2017-03-14 | Sybase, Inc. | Methods and systems for load balancing databases in a cloud environment |
-
2021
- 2021-02-23 CN CN202110204142.XA patent/CN112905345B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104580439A (en) * | 2014-12-30 | 2015-04-29 | 创新科存储技术（深圳）有限公司 | Method for achieving uniform data distribution in cloud storage system |
| CN109783000A (en) * | 2017-11-10 | 2019-05-21 | 成都华为技术有限公司 | A kind of data processing method and equipment |
| CN109144411A (en) * | 2018-07-24 | 2019-01-04 | 中国电子科技集团公司第三十八研究所 | Data center's hybrid magnetic disc array and its data dynamic migration strategy |
| CN111290699A (en) * | 2018-12-07 | 2020-06-16 | 杭州海康威视系统技术有限公司 | Data migration method, device and system |
| CN109992217A (en) * | 2019-04-11 | 2019-07-09 | 苏州浪潮智能科技有限公司 | A kind of method for controlling quality of service, device, electronic equipment and storage medium |
| CN112379825A (en) * | 2019-09-24 | 2021-02-19 | 北京城建设计发展集团股份有限公司 | Distributed data storage method and device based on data feature sub-pools |
| CN110764706A (en) * | 2019-10-25 | 2020-02-07 | 普联技术有限公司 | Storage system, data management method, and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| "区域化混合存储结构与优化技术研究";谢徐超;《中国博士学位论文全文数据库（电子期刊）信息科技辑》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112905345A (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10185592B2 (en) | Network storage device using dynamic weights based on resource utilization | |
| US9866450B2 (en) | Methods and apparatus related to management of unit-based virtual resources within a data center environment | |
| US9454407B2 (en) | Service resource allocation | |
| KR101540631B1 (en) | System, method and recording medium recording the program thereof for dynamic expansion of the virtual cluster | |
| US9552230B2 (en) | Apparatus and method for allocating a task based on measured and estimated core usage | |
| US9298512B2 (en) | Client placement in a computer network system using dynamic weight assignments on resource utilization metrics | |
| CN109976917B (en) | Load scheduling method, device, load scheduler, storage medium and system | |
| CN102541460B (en) | Multiple disc management method and equipment | |
| US20150196841A1 (en) | Load balancing system and method for rendering service in cloud gaming environment | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| WO2017020742A1 (en) | Load balancing method and device | |
| US20230195315A1 (en) | Nonvolatile memory scheduling method, system and device, and readable storage medium | |
| CN112905345B (en) | Task allocation method, distributed storage system and server | |
| CN115840649B (en) | Method and device for partitioning capacity block type virtual resource allocation, storage medium and terminal | |
| CN111510479A (en) | Resource allocation method and device for heterogeneous cache system | |
| CN109831391B (en) | Flow control method, storage device and system in distributed storage system | |
| CN116483553A (en) | Computing device, data processing method, system and related device | |
| CN114968073A (en) | Data prefetching method, equipment and system | |
| CN114489463A (en) | Method and device for dynamically adjusting QOS (quality of service) of storage volume and computing equipment | |
| CN108762679B (en) | Method for combining online DDP (distributed data processing) and offline DDP (distributed data processing) and related device thereof | |
| CN116991335B (en) | Queue resource allocation method and device, disk array card, equipment and medium | |
| CN117640541B (en) | Cloud server resource allocation method, device, equipment and medium | |
| CN114760327B (en) | Cloud disk resource allocation adjusting method and device | |
| CN116405500B (en) | System resource management method based on data analysis and cloud computing data analysis | |
| CN117640541A (en) | Cloud server resource allocation method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |