CN115373609A - Task processing method and related equipment - Google Patents

Task processing method and related equipment Download PDF

Info

Publication number
CN115373609A
CN115373609A CN202211306798.3A CN202211306798A CN115373609A CN 115373609 A CN115373609 A CN 115373609A CN 202211306798 A CN202211306798 A CN 202211306798A CN 115373609 A CN115373609 A CN 115373609A
Authority
CN
China
Prior art keywords
stripe
target
width
task
storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211306798.3A
Other languages
Chinese (zh)
Inventor
李辉
邸忠辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Electronic Information Industry Co Ltd
Original Assignee
Inspur Electronic Information Industry Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Electronic Information Industry Co Ltd filed Critical Inspur Electronic Information Industry Co Ltd
Priority to CN202211306798.3A priority Critical patent/CN115373609A/en
Publication of CN115373609A publication Critical patent/CN115373609A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671In-line storage system
    • G06F3/0683Plurality of storage devices
    • G06F3/0689Disk arrays, e.g. RAID, JBOD
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604Improving or facilitating administration, e.g. storage management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629Configuration or reconfiguration of storage systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to the technical field of cloud computing data centers, and discloses a task processing method and related equipment, wherein the method comprises the steps of obtaining the historical task size of each target storage strip in a disk array; calculating to obtain the average size of the historical tasks according to the size of each historical task; determining the width of a target strip according to the average size of the historical tasks; performing width division on each target storage stripe according to the target stripe width to obtain divided storage stripes; and executing the target task by utilizing the divided storage stripes. By applying the technical scheme provided by the application, the RAID stripes are reasonably arranged in width so as to deal with task scenes of different sizes.

Description

Task processing method and related equipment
Technical Field
The application relates to the technical field of cloud computing data centers, in particular to a task processing method and related equipment.
Background
In cloud computing data centers, data storage performance is a core concern for users. In a storage system, a RAID (Redundant Array of Independent disks) is divided according to a stripe, and a RAID stripe is divided according to each Disk block. The task of one RAID stripe (generally, IO task) is converted into a task of multiple disk blocks, and the task of the RAID stripe is completed after the task of all the disk blocks is completed.
However, after RAID is created, the sizes of stripes and blocks are usually fixed, and the sizes of IO in different scenarios are different, and if RAID stripe width is set to be large, when IO is a small IO scenario, each write IO needs to apply for a stripe lock or monopolize a lock, if small IO pressure is large, contention for a stripe lock may be frequent, and RAID performance may be poor; for medium IO, it is still treated as multiple small IO, which is much larger than the large IO write amplification of the full stripe. Conversely, if the RAID stripe width setting is small, the performance will also be poor when the IO is mainly a large IO, and meanwhile the smaller the RAID full stripe write width is, the larger its write amplification will be, which reduces the performance of the large IO. Generally, the size of the RAID stripe width needs to be determined according to the usage scenario of RAID, and if the small IO is dominant, the RAID stripe width may be set to be smaller, and if the large IO is dominant, the stripe width may be set to be larger. However, the IO scenario of the storage system may change, for example, a certain period of time is mainly small IO, and a certain period of time is mainly large IO; or some areas of RAID are mainly large IO and some areas are mainly small IO. Business processes with only large and small IOs are rare. The difficulty in configuring the width of the RAID stripe is brought, and the performance of large IO and small IO is difficult to be considered in practical operation, or the large IO is configured to have good performance and the small IO is configured to have poor performance; or the small IO performance is good and the large IO performance is poor; or a compromise is adopted, and a comparatively balanced but less than optimal configuration is adopted.
Therefore, how to set the width of a RAID stripe reasonably to cope with task scenes of different sizes is a problem to be solved urgently by those skilled in the art.
Disclosure of Invention
The task processing method can set the width of the RAID stripe reasonably so as to deal with task scenes of different sizes; another object of the present application is to provide a task processing device, an apparatus, and a computer-readable storage medium, all having the above-mentioned advantageous effects.
In a first aspect, the present application provides a task processing method, including:
acquiring the historical task size of each target storage strip in the disk array;
calculating to obtain the average size of the historical tasks according to the size of each historical task;
determining the width of a target strip according to the average size of the historical tasks;
dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes;
and executing the target task by using the divided storage stripes.
Optionally, the obtaining the historical task size of each target storage stripe in the disk array includes:
acquiring a task record of each target storage stripe in the disk array within a preset time period;
and determining the historical task size of each target storage strip according to the task record.
Optionally, the determining a target stripe width according to the average size of the historical tasks includes:
acquiring an initial stripe width of the target storage stripe;
and determining the width of the target band according to a comparison strategy between the average size of the historical tasks and the initial band width.
Optionally, the determining a target stripe width according to the average size of the historical tasks includes:
determining the width of a target band corresponding to the average size of the historical tasks according to a preset mapping relation; the preset mapping relation is the mapping relation between the average size of the historical tasks and the width of the target band.
Optionally, after the width division is performed on each target storage stripe according to the target stripe width to obtain a divided storage stripe, the method further includes:
determining the row number of an array row of each storage stripe in the disk array, and acquiring the maximum divisible row number of the row of the disk array;
for each said array row, calculating the product of said row number and said row maximum partitionable stripe number;
the product is taken as the stripe number of the first memory stripe in the array row, and the other memory stripes in the array row are numbered based on the stripe number.
Optionally, before obtaining the historical task size of each target storage stripe in the disk array, the method further includes:
creating the disk array according to configuration information; the configuration information includes the number of disks, the initial stripe width, and the size of the data blocks dividing the disks.
Optionally, after the width division is performed on each target storage stripe according to the target stripe width to obtain a divided storage stripe, the method further includes:
and converting the original data in each target storage stripe into the new data of each divided storage stripe.
Optionally, the converting the original data in each target storage stripe into new data of each divided storage stripe includes:
setting a data switching mark for each target storage stripe;
and when the task to be processed before the data switching mark is set is executed, converting the original data in each target storage stripe into the new data of each divided storage stripe.
Optionally, the task processing method further includes:
and storing the tasks to be processed after the data switching marks are set to a waiting linked list.
Optionally, after converting the original data in each target storage stripe into new data of each divided storage stripe, the method further includes:
deleting each data switching mark;
and reading the target task from the waiting linked list to execute the step of executing the target task by utilizing the divided storage strips.
Optionally, after the target task is executed by using the divided storage stripe, the method further includes:
and logging the task execution process of each storage stripe in the disk array.
Optionally, the task processing method further includes:
and restoring each storage stripe in the disk array to an initial state according to a restoring instruction.
In a second aspect, the present application further discloses a task processing device, the device comprising:
the acquisition module is used for acquiring the historical task size of each target storage strip in the disk array;
the calculation module is used for calculating and obtaining the average size of the historical tasks according to the size of each historical task;
the determining module is used for determining the width of a target strip according to the average size of the historical tasks;
the dividing module is used for dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes;
and the execution module is used for executing the target task by utilizing the divided storage strips.
In a third aspect, the present application further discloses a task processing device, including:
a memory for storing a computer program;
a processor for implementing the steps of any one of the task processing methods as described above when executing the computer program.
In a fourth aspect, the present application further discloses a computer readable storage medium having a computer program stored thereon, which when executed by a processor, implements the steps of any one of the task processing methods described above.
The task processing method provided by the application comprises the following steps: acquiring the historical task size of each target storage strip in the disk array; calculating to obtain the average size of the historical tasks according to the size of each historical task; determining the width of a target strip according to the average size of the historical tasks; dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes; and executing the target task by using the divided storage stripes.
By applying the technical scheme provided by the application, reasonable width change can be carried out on each storage strip in the disk array, and aiming at the target storage strip needing width change in the disk array, the proper target strip width can be determined according to the average size of the historical tasks processed by the storage strip width, so that width division can be carried out on each target storage strip again according to the target strip width, the purpose that the width of the target storage strip is changed from the initial strip width to the target strip width is achieved, reasonable adjustment of the storage strip width in the disk array is further achieved, the disk array can better cope with task scenes with different sizes, and the overall performance of the disk array is greatly improved.
The task processing device, the apparatus, and the computer-readable storage medium provided in the present application also have the above technical effects, and are not described herein again.
Drawings
In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings that are needed to be used in the description of the prior art and the embodiments of the present application will be briefly described below. Of course, the following description of the drawings related to the embodiments of the present application is only a part of the embodiments of the present application, and it will be obvious to those skilled in the art that other drawings can be obtained from the provided drawings without any creative effort, and the obtained other drawings also belong to the protection scope of the present application.
Fig. 1 is a schematic flowchart of a task processing method provided in the present application;
FIG. 2 is a schematic structural diagram of a disk array provided in the present application;
FIG. 3 is a schematic diagram of a data conversion system according to the present application;
FIG. 4 is a flowchart illustrating a task processing device according to the present application;
fig. 5 is a schematic structural diagram of a task processing device provided in the present application.
Detailed Description
The core of the application is to provide a task processing method, which can carry out reasonable width setting on the RAID stripes so as to deal with task scenes with different sizes; another core of the present application is to provide a task processing device, a device and a computer readable storage medium, all having the above beneficial effects.
In order to more clearly and completely describe the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
The embodiment of the application provides a task processing method.
Referring to fig. 1, fig. 1 is a flowchart illustrating a task processing method according to the present application, where the task processing method includes the following steps S101 to S105.
S101: acquiring the historical task size of each target storage strip in the disk array;
the method aims to achieve the purpose of obtaining the historical task size of each target storage stripe in the disk array. The disk array is used for realizing task processing; the target storage stripe refers to a storage stripe in the disk array, which needs to be subjected to stripe width change, and may be all storage stripes in the disk array or a part of storage stripes in the disk array, and may be specifically set by a technician according to actual needs, which is not limited in this application; the historical task size refers to the task size of each historical task processed by the corresponding target storage stripe, and is used for calculating the average historical task size.
It is understood that during the task processing, a log record is generally performed for recording log information generated during the task processing, including but not limited to a task type, a task size, task processing intermediate information, a task processing result, exception information or failure information, and the like. Based on this, the historical task size of each target storage stripe in the disk array can be extracted from the log file.
Therefore, in a possible implementation manner, the obtaining of the historical task size of each target storage stripe in the disk array may include the following steps:
acquiring task records of each target storage strip in the disk array within a preset time period;
and determining the historical task size of each target storage strip according to the task record.
Specifically, a task record of each target storage stripe in the disk array within a preset time period, that is, the log record, may be called first, where relevant information of each historical task processed by each target storage stripe within the preset time period is recorded, and therefore, the historical task size of each target storage stripe may be obtained through parsing from the task record.
As described above, the historical task size is used to implement calculation of the average historical task size, and therefore, in the embodiment of the present application, task records of each target storage stripe in a preset time period are selected and acquired, so as to calculate the average historical task size in the preset time period. It can be understood that, when the value of the preset time period is larger, the number of the obtained historical tasks is larger, which may cause the increase of the calculation cost and the occupation of the calculation resources; when the value of the preset time period is small, the number of the obtained historical tasks is small, and the problem that the calculation result is not accurate enough may be caused. Therefore, the value of the preset time period can be set through a large number of experiments, or the value of the preset time period can be set according to historical experience, for example, the value of the preset time period can be set to 24 hours.
S102: calculating to obtain the average size of the historical tasks according to the size of each historical task;
this step is intended to enable the calculation of the historical average task size. After the size of each historical task of each target storage strip is obtained, average calculation can be carried out to obtain the average size of the historical tasks, and the average size of the historical tasks is the average task size of the historical tasks processed by the target storage strips. For the implementation manner of obtaining the task record of each target storage stripe in the preset time period, the average size of the historical tasks is the average size of the tasks of each historical task processed by the target storage stripe in the preset time period. Further, when the number of target storage stripes is multiple, then the average size of the historical tasks of all target storage stripes may be calculated.
S103: determining the width of a target strip according to the average size of the historical tasks;
the step aims to determine a target stripe width, which is a stripe width that needs to be achieved after the target storage stripe is subjected to width change, so that the target storage stripe is subjected to width division again by using the target stripe width to obtain a storage stripe meeting the target stripe width. After the average size of the historical tasks of the target storage stripe is obtained, the width of the target stripe can be determined according to the historical average tasks, and the implementation process can be implemented based on a preset computing strategy.
For example, in a possible implementation manner, the determining the target stripe width according to the average size of the historical tasks may include the following steps:
acquiring the initial stripe width of a target storage stripe;
and determining the width of the target strip according to a comparison strategy between the average size of the historical tasks and the initial strip width.
The embodiment of the application provides an implementation method for determining the width of a target strip according to the average size of historical tasks, namely the implementation is realized based on a comparison strategy between the average size of the historical tasks and the width of an initial strip. First, an initial stripe width of a target storage stripe may be obtained, where the initial stripe width is a stripe width set when a disk array is created, and generally, the stripe width set when the disk array is created is an initial width of all storage stripes in the disk array, in other words, widths of storage stripes in the initially created disk array are uniform. Further, the target strip width can be determined by combining the comparison strategy between the average size of the historical tasks and the initial strip width. The following example provides an alignment strategy between the average size of the historical tasks and the initial stripe width:
when the average historical task size is less than or equal to twice the data block size, setting the target stripe width to be 2 (the two data block sizes are referred to herein); when the average size of the historical tasks is larger than twice the size of the data blocks and is smaller than or equal to four times the size of the data blocks, the target stripe width is set to be 4 (the size of the four data blocks is referred to herein); when the average historical task size is greater than four times the data chunk size and less than or equal to eight times the data chunk size, the target stripe width is set to 8 (which refers to eight data chunk sizes here), and so on. It should be noted that each disk in the disk array is divided into data blocks according to the size of the fixed block, where the data block size is the size of the data block after being partitioned.
For another example, in a possible implementation manner, the determining the target stripe width according to the average size of the historical tasks may include the following steps:
determining the width of a target band corresponding to the average size of the historical tasks according to a preset mapping relation; the preset mapping relation is the mapping relation between the average size of the historical tasks and the width of the target strip.
The embodiment of the application provides another implementation method for determining the width of a target band according to the average size of historical tasks, namely, the implementation is realized on the basis of a mapping relation between the average size of the historical tasks and the width of the target band which are created in advance. Specifically, a mapping relationship between the average size of the historical task and the target stripe width may be created in advance, that is, the preset mapping relationship is pre-stored in a preset storage space and can be directly called when in use, so that after the average size of the historical task is obtained through calculation, a value of the target stripe width corresponding to the average size of the historical task can be queried from the preset mapping relationship.
S104: dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes;
the step is intended to implement width division of the target storage stripe, and after determining the target stripe width of the target storage stripe, the width division may be performed again on each target storage stripe according to the target stripe width, that is, the width is changed to obtain the divided storage stripe. For example, if four target storage stripes with the required width are needed, the initial width of each target storage stripe is 2, and thus the total width of the four target storage stripes is 8, then when the target stripe width is 4, two divided storage stripes are provided; when the target stripe width is 8, there is only one divided storage stripe.
S105: and executing the target task by using the divided storage stripes.
This step is intended to achieve the execution of the target task. After the width of the target storage stripe is changed, the received target task, that is, the task to be processed that is newly received after the width of the target storage stripe is changed, may be continuously executed by directly using the divided storage stripes, and of course, the task type does not affect the implementation of the present technical solution, and may be applied to the task processing of the disk array, which is not limited in the present application.
Therefore, the task processing method provided by the embodiment of the application can reasonably change the width of each storage stripe in the disk array, and can determine the appropriate target stripe width according to the average size of the processed historical tasks aiming at the target storage stripe of which the width needs to be changed in the disk array, so that the width of each target storage stripe can be divided again according to the target stripe width, the purpose that the width of the target storage stripe is changed from the initial stripe width to the target stripe width is achieved, the reasonable adjustment of the width of the storage stripe in the disk array is further achieved, the disk array can better cope with task scenes of different sizes, and the overall performance of the disk array is greatly improved.
In an embodiment of the application, after the width division is performed on each target storage stripe according to the target stripe width to obtain the divided storage stripes, the method may further include the following steps:
determining the row number of an array row where each storage strip is located in the disk array, and acquiring the maximum divisible strip number of the row of the disk array;
for each array row, calculating the product of the row number and the maximum divisible strip number of the row;
the product is taken as the stripe number of the first memory stripe in the array row and the other memory stripes in the array row are numbered based on the stripe number.
According to the task processing method provided by the embodiment of the application, after the width of the target storage stripe is changed, the divided storage stripes can be further numbered again. It can be understood that, when the disk array is created, each storage stripe therein is provided with a unique number, so that after the width change of the target storage stripe is completed, the storage stripes in the disk array can be numbered again.
In implementation, for the disk array, a row number of an array row to which each divided storage stripe belongs may be determined, for example, when the row number is 0, the array row corresponds to a first array row, and when the row number is 1, the array row corresponds to a second array row; meanwhile, acquiring the maximum number of the divisible strips of the row of the disk array, namely the maximum number of the divisible storage strips in each array row when the disk array is initially established; further, calculating the product of the row number and the maximum divisible stripe number of the row, and taking the product as the stripe number of the first storage stripe in the current array row; finally, numbering other storage stripes in the current array row based on the stripe number of the first storage stripe, for example, when the product of the row number and the maximum divisible row number is 4, that is, the stripe number of the first storage stripe in the current array row is 4, the stripe number of the second storage stripe in the current array row is 5, the stripe number of the second storage stripe in the current array row is 6, and so on until all storage stripes in the current array row are numbered, and continuing the numbering of the next array row.
In an embodiment of the present application, before obtaining the historical task size of each target storage stripe in the disk array, the method may further include the following steps:
creating a disk array according to the configuration information; the configuration information includes the number of disks, the initial stripe width, and the size of the data blocks dividing the disks.
The task processing method provided by the embodiment of the application can further realize the creation of the disk array, and the task processing method can be directly realized according to the configuration information when the disk array is created. The configuration information is some parameters set by technicians according to actual requirements, including but not limited to the number of disks, the initial stripe width, and the size of data blocks for dividing the disks as described above, where the number of disks refers to the number of disks in the created disk array; the initial stripe width refers to the initial width of each storage stripe in the created disk array; the size of the data block of the divided disk is the size of the data block when the data block division is performed on the disk.
In an embodiment of the application, after the width division is performed on each target storage stripe according to the target stripe width to obtain the divided storage stripes, the method may further include the following steps:
and converting the original data in each target storage stripe into new data of each divided storage stripe.
After the stripe width changing operation is completed, the task processing method provided by the embodiment of the application may further perform a data conversion operation, that is, convert the original data in each target storage stripe into new data of each divided storage stripe. The data to be converted mainly includes stripe data and metadata in the storage stripe, the stripe data is data information for implementing task processing, and the metadata is data information for representing information such as stripe width and storage position.
In an embodiment of the present application, the converting the original data in each target storage stripe into new data of each divided storage stripe may include the following steps:
setting a data switching mark for each target storage stripe;
and when the task to be processed before the data switching flag is set is executed, converting the original data in each target storage stripe into the new data of each divided storage stripe.
The embodiment of the application provides an implementation method for converting original data in each target storage stripe into new data of each divided storage stripe. Firstly, when determining that data conversion is needed, setting a data switching mark for each target storage stripe needing data switching; further, when the data switching flag is set, the task processing device may be currently in a task processing state, that is, there are some to-be-processed tasks that have been received but have not been processed or have not started to be processed, and then, for the to-be-processed tasks received before the data switching flag is set, the task processing may be continued first until the to-be-processed tasks are executed completely, and then the step of converting the original data in each target storage stripe into new data of each divided storage stripe is continued.
In an embodiment of the present application, the task processing method may further include the steps of:
and storing the tasks to be processed after the data switching marks are set to a waiting linked list.
It can be understood that, during the process of executing the step of converting the original data in each target storage stripe into the new data of each divided storage stripe, the task processing device may receive a new task to be processed, and since the task processing device is currently in the data conversion state, if it is likely that an exception occurs if the task processing is performed simultaneously, all the tasks to be processed received after the data switching flag is set may be temporarily stored in the waiting list, and the task processing may be continued until the data conversion is completed. In combination with the above embodiment, it can be seen that, for a task to be processed received before performing a data conversion operation, the task processing is performed first, and then the data conversion operation is performed; for the to-be-processed tasks received in the process of executing the data conversion operation, the to-be-processed tasks are firstly suspended, the data conversion operation is continuously executed, and then the task processing is continuously executed after the data conversion operation is completed.
In an embodiment of the application, after the converting the original data in each target storage stripe into the new data of each divided storage stripe, the method may further include the following steps:
deleting each data switching mark;
and reading the target task from the waiting linked list to execute the step of executing the target task by utilizing the divided storage strips.
Specifically, after the data conversion operation is completed, the data switching flag in the target storage stripe may be further deleted, and the subsequent task processing operation may be continued. At this time, each target task can be directly read from the waiting linked list, and task processing is sequentially performed. Of course, since each task to be processed in the waiting list needs to be processed first, the task to be processed newly received in the process can be temporarily stored in the waiting list, that is, each task to be processed is sequentially executed according to the sequence of the received tasks to be processed.
In an embodiment of the present application, after the target task is executed by using the divided storage stripe, the method may further include the following steps:
and logging the task execution process of each storage stripe in the disk array.
As described above, the historical task size of each target storage stripe in the disk array may be extracted from the log file, and based on this, after the target task is executed by using the divided storage stripes, the task execution process of each storage stripe in the disk array may be logged, where the content to be recorded mainly includes the task size of the target task, so as to provide reference data for a subsequent new storage stripe width change operation. Of course, the content to be logged is not limited to the size of the target task, and may also include other data information, such as task type, task processing intermediate information, task processing result, exception information, or fault information.
In an embodiment of the present application, the task processing method may further include the steps of:
and restoring each storage stripe in the disk array to an initial state according to the restoring instruction.
The task processing method provided by the embodiment of the application can further realize the restoration of the width of the storage stripe on the basis of realizing the change of the width of the storage stripe, and the restoration operation of the width of the storage stripe can be realized on the basis of the restoration instruction. Specifically, when a recovery requirement occurs, a technician may initiate a recovery instruction through the terminal device, so that the task processing device may respond to the recovery instruction to recover the storage stripes in the disk array to an initial state, where recovering the storage stripes in the disk array to the initial state may include recovering the stripe widths of the storage stripes in the disk array to the initial stripe widths, and recovering the data of the storage stripes in the disk array to the original data before the stripe widths are changed. Therefore, restoration of each storage stripe in the disk array is realized.
The embodiment of the application provides another task processing method.
First, the stripe width of the storage stripe in RAID (here, the stripe width refers to the number of data blocks included in one storage stripe) may be firstly divided into levels, for example, the storage stripe may be divided into three levels, such as 4, 8, and 16, or three levels, such as 3, 6, and 12, or four levels, such as 2, 4, 8, and 16, according to a common ratio of 2, and specific values may be set accordingly according to a specific configuration of the storage RAID. For convenience of description, the following explanation is made using three levels of 2, 4, and 8.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a disk array (hereinafter referred to as RAID) provided by the present application, where the disk array includes 12 disks, each disk is divided into data blocks according to a fixed block size, a first block of all the disks forms a first row, a second block of all the disks forms a second row, and widths of all stripes in the same row are the same. Then the user can either, for example,
if a storage stripe contains 2 data blocks and one parity block, then a row of disk blocks may be divided into four storage stripes;
if a storage stripe contains 4 data blocks and a parity block, then a row of disk blocks may be divided into two storage stripes, with each storage stripe followed by a spare block after the parity block;
if a storage stripe contains 8 data blocks and one parity block, then a row of disk blocks may be divided into one storage stripe with each storage stripe followed by three spare blocks after the parity block.
For each row of memory stripes, the stripe numbers start with (row number x row maximum number of partitionable stripes), in this example the row maximum number of partitionable stripes is 4, and the numbering of each memory stripe is as shown in fig. 2.
Further, the stripe width of the RAID may be set to its initial value when creating the RAID, and the stripe width of all or part of the storage stripes in the RAID after creation may be automatically changed in size according to the task situation. When creating RAID, the width of a storage stripe in RAID may be set according to a main service type, as in RAID shown in fig. 2, if a service is mainly a small block IO, the width of a stripe may be set to 2, and thus each row includes four stripes.
Then, the stripe width required to be set, i.e. the target stripe width, is automatically calculated according to the IO. RAID may record the IO size of each storage stripe and calculate an average value in a set period, for example, every 24 hours may be set as one calculation period, and RAID may automatically select the stripe data width according to the average size of the service IO in the past 24 hours.
Since the widths of all the memory stripes in the same row are the same in this embodiment, a certain memory stripe or all the memory stripes in the row may be sampled for calculation, wherein, to reduce the calculation amount, the first memory stripe in each row may be sampled, the target data width thereof may be calculated, and set as the target stripe width of the row. Wherein, the selection strategy is as follows:
(1) Calculating the target data width of the first stripe of a certain row:
when the average value is less than or equal to the data block size x 2, the target stripe width is set to 2;
when the average value is larger than the data block size x 2 and is smaller than or equal to the data block size x 4, the target stripe width is set to 4;
when the average value is larger than the data block size 4 and smaller than or equal to the data block size 8, the target stripe width is set to 8;
and so on.
2) The target data width of the first memory stripe of the row is set to the target data width of the row.
Finally, after the stripe data width of a certain row of storage stripes is reset, corresponding conversion needs to be performed on the stripe data and the metadata of the current row, that is, the data and the metadata of the original stripe width of each row are converted into the data and the metadata of the new stripe width. Before conversion, a state conversion flag string _ CHANGE may be set for the row storage stripe, and the issued IO before adding the state conversion flag may continue to be executed, and data conversion may be performed after the IO is executed; the IO issued after the state transition mark is added can be hung in a waiting chain table to wait, and the task processing is continued after the data transition is finished.
Referring to fig. 3, fig. 3 is a schematic diagram of data conversion provided by the present application, taking the first row of memory stripes in fig. 2 as an example of converting from the initial stripe width 2 to the target stripe width 4, the conversion process is as follows:
the method comprises the following steps: setting the state transition marks STRIDE _ CHANGE of all storage stripes in the first row, continuously executing and completely finishing all issued IOs in the first row, and if the issued IOs exist, hanging the wait _ q waiting linked list for waiting;
step two: repartitioning the first row into two storage stripes and designating a corresponding partition of each storage stripe with new metadata;
step three: the data blocks of the original stripe 0 and the original stripe 1 are sequentially placed into the corresponding data blocks of the new stripe, and the positions of D0_0 and D0_1 of the original stripe 0 and the positions of D0_0 and D0_1 of the new stripe 0 are completely the same, and the positions of D2_0 and D2_1 of the original stripe 2 and the positions of D1_0 and D1_1 of the new stripe 1 are completely the same, so that the data blocks can not be copied;
step four: recalculating the check data P0_ NEW of the NEW stripe 0 and the check data P1_ NEW of the NEW stripe 1, and writing the recalculated check data into a disk;
step five: and taking out the IO in the wait chain table to continue execution.
Therefore, the task processing method provided by the embodiment of the application can reasonably change the width of each storage stripe in the disk array, and can determine the appropriate target stripe width according to the average size of the processed historical tasks for the target storage stripe of which the width needs to be changed in the disk array, so that the width of each target storage stripe can be divided again according to the target stripe width, the width of the target storage stripe is changed from the initial stripe width to the target stripe width, the reasonable adjustment of the width of the storage stripe in the disk array is further realized, the disk array can better cope with task scenes with different sizes, and the overall performance of the disk array is greatly improved.
The embodiment of the application provides a task processing device.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a task processing device according to the present application, where the task processing device includes:
the acquisition module 1 is used for acquiring the historical task size of each target storage strip in the disk array;
the calculation module 2 is used for calculating and obtaining the average size of the historical tasks according to the size of each historical task;
the determining module 3 is used for determining the width of a target strip according to the average size of the historical tasks;
the dividing module 4 is used for dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes;
and the execution module 5 is used for executing the target task by using the divided storage stripes.
Therefore, the task processing device provided by the embodiment of the application can reasonably change the width of each storage stripe in the disk array, and can determine the proper target stripe width according to the average size of the processed historical tasks aiming at the target storage stripe in the disk array needing width change, so that width division is performed on each target storage stripe again according to the target stripe width, the purpose that the width of the target storage stripe is changed from the initial stripe width to the target stripe width is achieved, reasonable adjustment of the width of the storage stripe in the disk array is further achieved, the disk array can better cope with task scenes with different sizes, and the overall performance of the disk array is greatly improved.
In an embodiment of the present application, the obtaining module 1 may be specifically configured to obtain task records of each target storage stripe in a disk array within a preset time period; and determining the historical task size of each target storage strip according to the task record.
In an embodiment of the present application, the determining module 3 may be specifically configured to obtain an initial stripe width of a target storage stripe; and determining the width of the target strip according to a comparison strategy between the average size of the historical tasks and the initial strip width.
In an embodiment of the present application, the determining module 3 may be specifically configured to determine a target stripe width corresponding to an average size of the historical task according to a preset mapping relationship; the preset mapping relation is the mapping relation between the average size of the historical tasks and the width of the target strip.
In an embodiment of the present application, the task processing device may further include a numbering module, configured to determine a row number of an array row in which each storage stripe is located in the disk array after the target storage stripes are subjected to width division according to the target stripe width to obtain divided storage stripes, and obtain a maximum number of partitionable stripes in the row of the disk array; for each array row, calculating the product of the row number and the maximum divisible strip number of the row; the product is taken as the stripe number of the first memory stripe in the array row and the other memory stripes in the array row are numbered based on the stripe number.
In an embodiment of the present application, the task processing apparatus may further include a creating module, configured to create the disk array according to the configuration information before the obtaining of the historical task size of each target storage stripe in the disk array; the configuration information includes the number of disks, the initial stripe width, and the size of the data blocks dividing the disks.
In an embodiment of the present application, the task processing device may further include a conversion module, configured to, after the width division is performed on each target storage stripe according to the target stripe width and the divided storage stripes are obtained, convert original data in each target storage stripe into new data of each divided storage stripe.
In an embodiment of the present application, the conversion module may be specifically configured to set a data switching flag for each target storage stripe; and when the task to be processed before the data switching mark is set is completely executed, converting the original data in each target storage stripe into the new data of each divided storage stripe.
In an embodiment of the present application, the task processing apparatus may further include a storage module, configured to store the to-be-processed task after the data switch flag is set to the waiting list.
In an embodiment of the present application, the task processing device may further include a reading module, configured to delete each data switching flag after the original data in each target storage stripe is converted into new data of each divided storage stripe; and reading the target task from the waiting linked list to execute the step of executing the target task by utilizing the divided storage strips.
In an embodiment of the present application, the task processing device may further include a logging module, configured to log a task execution process of each storage stripe in the disk array after the target task is executed by using the divided storage stripes.
In an embodiment of the application, the task processing device may further include a recovery module, configured to recover the storage stripes in the disk array to an initial state according to the recovery instruction.
For the introduction of the apparatus provided in the embodiment of the present application, please refer to the method embodiment described above, which is not described herein again.
The embodiment of the application provides a task processing device.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a task processing device provided in the present application, where the task processing device may include:
a memory for storing a computer program;
a processor, configured to implement the steps of any one of the task processing methods described above when executing the computer program.
As shown in fig. 5, in order to illustrate the structure of the task processing device, the task processing device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through a communication bus 13.
In the embodiment of the present application, the processor 10 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, etc.
The processor 10 may call a program stored in the memory 11, and in particular, the processor 10 may perform operations in an embodiment of the task processing method.
The memory 11 is used for storing one or more programs, the program may include program codes, the program codes include computer operation instructions, in this embodiment, the memory 11 stores at least the program for implementing the following functions:
acquiring the historical task size of each target storage strip in the disk array;
calculating to obtain the average size of the historical tasks according to the size of each historical task;
determining the width of a target strip according to the average size of the historical tasks;
dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes;
and executing the target task by using the divided storage stripes.
In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created during use.
Further, the memory 11 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.
The communication interface 12 may be an interface of a communication module for connecting with other devices or systems.
Of course, it should be noted that the structure shown in fig. 5 does not constitute a limitation to the task processing device in the embodiment of the present application, and in practical applications, the task processing device may include more or less components than those shown in fig. 5, or some components may be combined.
The embodiment of the application provides a computer readable storage medium.
The computer-readable storage medium provided in the embodiments of the present application stores a computer program, and the computer program, when executed by a processor, can implement the steps of any one of the task processing methods described above.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For introduction of the computer-readable storage medium provided in the embodiment of the present application, please refer to the method embodiment described above, which is not described herein again.
The embodiments are described in a progressive mode in the specification, the emphasis of each embodiment is on the difference from the other embodiments, and the same and similar parts among the embodiments can be referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The technical solutions provided in the present application are described in detail above. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principle of the present application, the present application can also make several improvements and modifications, and those improvements and modifications also fall into the protection scope of the present application.

Claims (15)

1. A method for processing a task, the method comprising:
acquiring the historical task size of each target storage strip in the disk array;
calculating to obtain the average size of the historical tasks according to the size of each historical task;
determining the width of a target strip according to the average size of the historical tasks;
dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes;
and executing the target task by using the divided storage stripes.
2. The method of claim 1, wherein the obtaining the historical task size of each target storage stripe in the disk array comprises:
acquiring a task record of each target storage stripe in the disk array within a preset time period;
and determining the historical task size of each target storage stripe according to the task record.
3. The method of claim 1, wherein determining a target stripe width based on the historical task average size comprises:
acquiring an initial stripe width of the target storage stripe;
and determining the width of the target band according to a comparison strategy between the average size of the historical tasks and the width of the initial band.
4. The method of claim 1, wherein determining a target stripe width based on the historical task average size comprises:
determining the width of a target band corresponding to the average size of the historical tasks according to a preset mapping relation; the preset mapping relation is the mapping relation between the average size of the historical tasks and the width of the target band.
5. The method of claim 1, wherein after the width-dividing each target storage stripe according to the target stripe width to obtain the divided storage stripes, further comprising:
determining the row number of an array row of each storage stripe in the disk array, and acquiring the maximum divisible row number of the row of the disk array;
for each said array row, calculating the product of said row number and said row maximum partitionable stripe number;
the product is taken as the stripe number of the first memory stripe in the array row, and the other memory stripes in the array row are numbered based on the stripe number.
6. The method of claim 1, wherein before obtaining the historical task size of each target storage stripe in the disk array, further comprising:
creating the disk array according to configuration information; the configuration information includes the number of disks, the initial stripe width and the size of the data blocks for dividing the disks.
7. The method according to any one of claims 1 to 6, wherein after the width dividing is performed on each target storage stripe according to the target stripe width, and obtaining divided storage stripes, the method further comprises:
and converting the original data in each target storage stripe into the new data of each divided storage stripe.
8. The method of claim 7, wherein converting the original data in each of the target memory stripes into new data for each of the divided memory stripes comprises:
setting a data switching mark for each target storage stripe;
and when the task to be processed before the data switching mark is set is executed, converting the original data in each target storage stripe into the new data of each divided storage stripe.
9. The method of claim 8, further comprising:
and storing the tasks to be processed after the data switching marks are set to a waiting linked list.
10. The method of claim 9, wherein after converting the original data in each of the target memory stripes into new data for each of the divided memory stripes, further comprising:
deleting each data switching mark;
and reading the target task from the waiting linked list to execute the step of executing the target task by utilizing the divided storage strips.
11. The method of claim 1, wherein after executing the target task using the partitioned memory stripe, further comprising:
and logging the task execution process of each storage stripe in the disk array.
12. The method of claim 1, further comprising:
and restoring each storage stripe in the disk array to an initial state according to a restoring instruction.
13. A task processing apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring the historical task size of each target storage strip in the disk array;
the calculation module is used for calculating and obtaining the average size of the historical tasks according to the size of each historical task;
the determining module is used for determining the width of a target strip according to the average size of the historical tasks;
the dividing module is used for dividing the width of each target storage stripe according to the width of the target stripe to obtain divided storage stripes;
and the execution module is used for executing the target task by utilizing the divided storage strips.
14. A task processing device characterized by comprising:
a memory for storing a computer program;
a processor for implementing the steps of the task processing method according to any one of claims 1 to 12 when executing the computer program.
15. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the task processing method according to any one of claims 1 to 12.
CN202211306798.3A 2022-10-25 2022-10-25 Task processing method and related equipment Pending CN115373609A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211306798.3A CN115373609A (en) 2022-10-25 2022-10-25 Task processing method and related equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211306798.3A CN115373609A (en) 2022-10-25 2022-10-25 Task processing method and related equipment

Publications (1)

Publication Number Publication Date
CN115373609A true CN115373609A (en) 2022-11-22

Family

ID=84073242

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211306798.3A Pending CN115373609A (en) 2022-10-25 2022-10-25 Task processing method and related equipment

Country Status (1)

Country Link
CN (1) CN115373609A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115543216A (en) * 2022-11-29 2022-12-30 苏州浪潮智能科技有限公司 Disk array distribution architecture, block calculation method, system, device and medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103268204A (en) * 2013-06-08 2013-08-28 北京百度网讯科技有限公司 Adjusting and optimizing method and device of solid-state disk
CN110058792A (en) * 2018-01-18 2019-07-26 伊姆西Ip控股有限责任公司 Expand method, equipment and the computer program product of memory space
CN110308875A (en) * 2019-06-27 2019-10-08 深信服科技股份有限公司 Data read-write method, device, equipment and computer readable storage medium
CN111124249A (en) * 2018-10-30 2020-05-08 伊姆西Ip控股有限责任公司 Method, apparatus and computer program product for data processing
CN113867644A (en) * 2021-09-30 2021-12-31 苏州浪潮智能科技有限公司 Disk array optimization method and device, computer equipment and storage medium
CN115129265A (en) * 2022-09-01 2022-09-30 苏州浪潮智能科技有限公司 Partitioned caching method, device, equipment and readable medium for redundant array of independent disks

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103268204A (en) * 2013-06-08 2013-08-28 北京百度网讯科技有限公司 Adjusting and optimizing method and device of solid-state disk
CN110058792A (en) * 2018-01-18 2019-07-26 伊姆西Ip控股有限责任公司 Expand method, equipment and the computer program product of memory space
CN111124249A (en) * 2018-10-30 2020-05-08 伊姆西Ip控股有限责任公司 Method, apparatus and computer program product for data processing
CN110308875A (en) * 2019-06-27 2019-10-08 深信服科技股份有限公司 Data read-write method, device, equipment and computer readable storage medium
CN113867644A (en) * 2021-09-30 2021-12-31 苏州浪潮智能科技有限公司 Disk array optimization method and device, computer equipment and storage medium
CN115129265A (en) * 2022-09-01 2022-09-30 苏州浪潮智能科技有限公司 Partitioned caching method, device, equipment and readable medium for redundant array of independent disks

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115543216A (en) * 2022-11-29 2022-12-30 苏州浪潮智能科技有限公司 Disk array distribution architecture, block calculation method, system, device and medium

Similar Documents

Publication Publication Date Title
WO2017107900A1 (en) Virtual machine recovery method and virtual machine management device
CN112463022B (en) IO processing method, device and medium based on disk array
WO2018006721A1 (en) Method and apparatus for storing log file
EP3474143B1 (en) Method and apparatus for incremental recovery of data
CN112256656B (en) Transaction rollback method and device, database, system and computer storage medium
CN110147203B (en) File management method and device, electronic equipment and storage medium
CN115373609A (en) Task processing method and related equipment
CN112182010A (en) Dirty page refreshing method and device, storage medium and electronic equipment
CN112463880B (en) Block chain data storage method and related device
CN109426587B (en) Data recovery method and device
CN108306780B (en) Cloud environment-based virtual machine communication quality self-optimization system and method
CN111459399A (en) Data writing method, data reading method and device
CN112988880B (en) Data synchronization method, device, electronic equipment and computer storage medium
CN115878052B (en) RAID array inspection method, inspection device and electronic equipment
CN115543871B (en) Data storage method and related equipment
CN106708445A (en) Link selection method and device
CN112506698A (en) Small file data reconstruction recovery method, system, terminal and storage medium
CN109960474A (en) Data-updating method, device, equipment and medium based on automatic simplify configuration
CN105573862B (en) Method and equipment for recovering file system
CN111880735A (en) Data migration method, device, equipment and storage medium in storage system
CN110703988B (en) Storage pool creating method, system, terminal and storage medium for distributed storage
CN112948173A (en) Data recovery method, device, equipment and medium
CN110688258A (en) Snapshot creating method and device
CN111026720A (en) File processing method, system and related equipment
CN111008195A (en) Database free space management method, system, terminal and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20221122