CN112463030B - Task processing method, device and system - Google Patents
Task processing method, device and system Download PDFInfo
- Publication number
- CN112463030B CN112463030B CN202011231202.9A CN202011231202A CN112463030B CN 112463030 B CN112463030 B CN 112463030B CN 202011231202 A CN202011231202 A CN 202011231202A CN 112463030 B CN112463030 B CN 112463030B
- Authority
- CN
- China
- Prior art keywords
- write
- storage
- volumes
- volume
- file read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 38
- 230000002776 aggregation Effects 0.000 claims abstract description 21
- 238000004220 aggregation Methods 0.000 claims abstract description 21
- 238000013507 mapping Methods 0.000 claims abstract description 17
- 230000004931 aggregating effect Effects 0.000 claims abstract description 12
- 238000000034 method Methods 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0665—Virtualisation aspects at area level, e.g. provisioning of virtual or logical volumes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The embodiment of the invention discloses a task processing method, a task processing device and a task processing system.A plurality of storage volumes are established in a storage pool of a storage end; establishing a plurality of network paths with a storage end according to the number of preset links; the SAN mapped storage volumes are resolved into block devices according to SAN mappings for the plurality of storage volumes included in each network path. According to a multipath aggregation mechanism, aggregating block devices mapped on different network paths by the same storage volume into a multipath device; each multipath device is initialized to a PV volume, and all PV volumes are combined into a virtual storage pool. Setting the strip width according to the size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, the file read-write task is split according to the stripe width and then is issued to a target PV volume of the virtual storage pool, and multi-path parallel processing of the file read-write subtasks is achieved according to a network path corresponding to the target PV volume, so that the processing efficiency of the file read-write task is improved.
Description
Technical Field
The present invention relates to the field of storage system technologies, and in particular, to a method, an apparatus, and a system for processing a task.
Background
In the application of the storage system, the read-write bit rate and the I/O request rate are two important indexes. Particularly in high-throughput application fields such as big data and cloud computing, the system performance greatly depends on the read-write rate of the storage system. Therefore, optimizing the read-write performance of a storage system is a major issue for storage device developers and solution designers.
In the traditional technology, the read-write performance of the storage system is optimized by increasing a transmission thread. However, the performance requirement of the storage system is higher as the number of the transmission threads is larger, and the number of the transmission threads which can be supported by the storage system is limited based on the processing capacity of the current storage system. And the control management of a large number of transmission threads also occupies a large amount of system resources, so that the mode of increasing the transmission threads is not obvious for improving the processing efficiency of the read-write task.
Therefore, how to effectively improve the processing efficiency of the file reading and writing task is a problem to be solved by the technical personnel in the field.
Disclosure of Invention
The embodiment of the invention aims to provide a task processing method, a task processing device and a task processing system, which can effectively improve the processing efficiency of a file reading and writing task.
To solve the foregoing technical problem, an embodiment of the present invention provides a task processing method, including:
creating a plurality of storage volumes in a storage pool of a storage side;
establishing a plurality of network paths with the storage end according to the preset number of links;
analyzing the SAN-mapped storage volumes into block devices according to SAN mapping of the storage volumes contained in each network path;
according to a multipath aggregation mechanism, aggregating block devices mapped on different network paths by the same storage volume into a multipath device; and initializing each of the multi-path devices into a PV volume; combining all the PV volumes into a virtual storage pool;
setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool;
when a file read-write task is received, splitting the file read-write task according to the stripe width and then sending the split file read-write task to a target PV volume of the virtual storage pool, and realizing multi-path parallel processing on a file read-write sub-task according to a network path corresponding to the target PV volume.
Optionally, the method includes, for capacity setting of a plurality of storage volumes in the storage side:
and setting the capacity of each storage volume according to the total capacity of the file system and the number of the storage volumes contained in the storage end.
Optionally, when receiving a file read-write task, splitting the file read-write task according to the stripe width and then sending the split file read-write task to the target PV volume in the virtual storage pool includes:
when a file reading and writing task is received, splitting the file reading and writing task into a plurality of file reading and writing subtasks according to the strip width;
when the number of the file read-write subtasks is the same as that of the PV volumes contained in the virtual storage pool, issuing a corresponding file read-write subtask to each PV volume;
when the number of the file read-write subtasks is smaller than the number of the PV volumes contained in the virtual storage pool, selecting a plurality of target PV volumes with the same number as the file read-write subtasks from the virtual storage pool, and issuing a corresponding file read-write subtask to each target PV volume;
and when the number of the file read-write sub-tasks is larger than that of the PV volumes contained in the virtual storage pool, issuing at least one file read-write sub-task to each PV volume.
The embodiment of the invention also provides a task processing device, which comprises a creating unit, an establishing unit, an analyzing unit, an aggregating unit, a setting unit and a splitting unit;
the creating unit is used for creating a plurality of storage volumes in a storage pool of a storage end;
the establishing unit is used for establishing a plurality of network paths with the storage end according to the preset number of links;
the analysis unit is configured to analyze the SAN-mapped storage volume into a block device according to the SAN mapping of the plurality of storage volumes included in each network path
The aggregation unit is used for mapping and aggregating the same storage volume on different network paths into a multi-path device according to a multi-path aggregation mechanism; and initializing all the multipath devices to PV volumes; combining all the PV volumes into a virtual storage pool;
the setting unit is used for setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool;
and the splitting unit is used for splitting the file reading and writing task according to the stripe width and then sending the split file reading and writing task to the target PV volume of the virtual storage pool when the file reading and writing task is received, and realizing multi-path parallel processing on the file reading and writing subtasks according to a network path corresponding to the target PV volume.
Optionally, the apparatus includes a setting unit configured to set a capacity of a plurality of storage volumes in the storage end;
the setting unit is used for setting the capacity of each storage volume according to the total capacity of the file system and the number of the storage volumes contained in the storage end.
Optionally, the splitting unit is specifically configured to split the file read-write task into a plurality of file read-write subtasks according to the stripe width when the file read-write task is received; when the number of the file read-write subtasks is the same as that of the PV volumes contained in the virtual storage pool, issuing a corresponding file read-write subtask to each PV volume; when the number of the file read-write subtasks is smaller than the number of the PV volumes contained in the virtual storage pool, selecting a plurality of target PV volumes with the same number as the file read-write subtasks from the virtual storage pool, and issuing a corresponding file read-write subtask to each target PV volume; and when the number of the file read-write subtasks is larger than that of the PV volumes contained in the virtual storage pool, issuing at least one file read-write subtask to each PV volume.
The embodiment of the invention also provides a task processing system, which comprises a storage end and a host end;
a plurality of storage volumes are created in the storage pool of the storage end;
the host end is used for establishing a plurality of network paths with the storage end according to the preset number of links; analyzing the SAN-mapped storage volumes into block devices according to SAN mapping of the storage volumes contained in each network path; according to a multipath aggregation mechanism, aggregating block devices mapped on different network paths by the same storage volume into a multipath device; and initializing each of the multi-path devices into a PV volume; combining all of the PV volumes into a virtual storage pool; setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, splitting the file read-write task according to the stripe width and then sending the split file read-write task to a target PV volume of the virtual storage pool, and realizing multi-path parallel processing on the file read-write subtasks according to a network path corresponding to the target PV volume.
According to the technical scheme, a plurality of storage volumes are created in the storage pool of the storage end; establishing a plurality of network paths with a storage end according to the number of preset links; and analyzing the SAN mapped storage volumes into block equipment according to the SAN mapping of the storage volumes contained in each network path, wherein the parallelization degree of the host end is the product of the number of the network paths and the number of the storage volumes. In order to reduce the management difficulty of a host end, block devices mapped by the same storage volume on different network paths can be aggregated into a multi-path device according to a multi-path aggregation mechanism; and initializing each multi-path device into a PV volume, and combining all PV volumes into a virtual storage pool, wherein at the moment, the host end only needs to manage and maintain a plurality of PV volumes in the virtual storage pool. Setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, the file read-write task is split according to the width of the strip and then is issued to a target PV volume of the virtual storage pool, and multi-path parallel processing of the file read-write subtasks is achieved according to a network path corresponding to the target PV volume. In the technical scheme, the file reading and writing tasks are sequentially subjected to LVM striping and SAN multipath transmission from the file system of the application end to the physical storage device, and are dispersed to a plurality of PV volumes and a plurality of network paths, so that the parallelization degree of the file system is greatly improved, and the processing efficiency of the file reading and writing tasks is effectively improved.
Drawings
In order to illustrate the embodiments of the present invention more clearly, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a task processing method according to an embodiment of the present invention;
FIG. 2 is an architecture diagram of a multi-volume aggregate file system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a task processing device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a task processing system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative work belong to the protection scope of the present invention.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Next, a task processing method provided in an embodiment of the present invention is described in detail. Fig. 1 is a flowchart of a task processing method according to an embodiment of the present invention, where the method includes:
s101: a plurality of storage volumes are created in a storage pool on a storage side.
In practical application, a plurality of storage volumes can be set at a storage end according to business requirements.
The capacity of each storage volume can be set according to the total capacity of the file system and the number of storage volumes included in the storage side. Assuming that the total capacity of the file system is T and the number of the storage volumes is N, the capacity of each storage volume may be set to T/N.
S102: and establishing a plurality of network paths with a storage end according to the preset number of links.
The file system comprises a storage end and a host end, wherein a plurality of network paths can be established between the host end and the storage end so as to facilitate the host end to call the storage volume in the storage end. In practical application, the number of links may be preset, so that the host may establish a corresponding number of network paths with the storage according to the number of links.
S103: the SAN mapped storage volumes are resolved into block devices according to SAN mappings for the plurality of storage volumes included in each network path.
The SAN maps for all storage volumes may be contained in each network path.
For example, assume that there are M network paths and N storage volumes, and each network path includes SAN maps of all the storage volumes, that is, each network path includes information of the N storage volumes, and at this time, the host may resolve to M × N block devices.
S104: according to a multipath aggregation mechanism, aggregating block devices mapped on different network paths by the same storage volume into a multipath device; initializing each multipath device into a PV volume; all PV volumes are combined into a virtual storage pool.
In order to reduce the management difficulty of a host end, the same storage volume can be mapped and aggregated into a multi-path device on different network paths according to a multi-path aggregation mechanism; initializing all multipath equipment into a PV volume; all PV volumes are combined into a virtual storage pool, at which time the host side only needs to manage and maintain multiple PV volumes.
Fig. 2 is an architecture diagram of a multi-volume aggregation file system according to an embodiment of the present invention, where 4 storage volumes and 4 network paths are taken as examples in fig. 2, and 1 network path is 1 session, and each session includes SAN maps of the 4 storage volumes. The mapping of the 4 storage volumes corresponding to each session is shown in fig. 2 in the form of a block device (sd), and different numbers are used in fig. 2 to distinguish different sds. Each session contains 4 block devices and 4 sessions contain 16 block devices in total.
In order to reduce the management difficulty of the host side, a multipath aggregation mode can be adopted to map and aggregate the same storage volume on different paths to restore the same storage volume to a multipath (multipath) device, so that each storage volume has a unique corresponding mpath device on the host side, and 4 mpath devices can be obtained through the multipath aggregation mode. In a specific implementation, a Logical Volume Manager (LVM) layer of the host may initialize an mpath device of each storage Volume as a PV Volume, and combine the PV volumes of 4 storage volumes into one virtual storage pool, that is, a VG pool, where the capacity of the VG pool is the capacity of the created file system.
S105: and setting the stripe width according to the preset service read-write block size and the number of PV volumes contained in the virtual storage pool.
The preset size of the service read-write block can be set according to the read-write data volume of the file read-write task in the conventional operation.
In practical application, the size of the preset service read-write block may be divided by the number of PV volumes included in the virtual storage pool, and the obtained quotient value is used as the stripe width.
In conjunction with the architecture of the multi-volume aggregate file system shown in FIG. 2, setting the stripe width can be viewed as creating a striped LV volume, and finally creating a file system on the LV volume and configuring the traffic.
S106: when a file read-write task is received, the file read-write task is split according to the width of the strip and then is issued to a target PV volume of the virtual storage pool, and multi-path parallel processing of the file read-write subtasks is achieved according to a network path corresponding to the target PV volume.
In the embodiment of the invention, when the file reading and writing task is received, the file reading and writing task can be split into a plurality of file reading and writing subtasks according to the strip width; and issuing each file read-write subtask to a target PV volume.
In practical application, when the number of the file read-write subtasks is the same as the number of the PV volumes included in the virtual storage pool, a corresponding file read-write subtask may be issued to each PV volume. And when the number of the file read-write subtasks is smaller than that of the PV volumes contained in the virtual storage pool, selecting a plurality of target PV volumes with the same number as the file read-write subtasks from the virtual storage pool, and issuing a corresponding file read-write subtask to each target PV volume. And when the number of the file read-write subtasks is larger than that of the PV volumes contained in the virtual storage pool, issuing at least one file read-write subtask to each PV volume.
Taking the example that the number of the file read-write subtasks is the same as the number of the PV volumes included in the virtual storage pool, assuming that the number of the PV volumes included in the virtual storage pool is 4, the file read-write task is split into 4 file read-write subtasks according to the stripe width, which are the file read-write subtask 1, the file read-write subtask 2, the file read-write subtask 3, and the file read-write subtask 4, respectively, at this time, the file read-write subtask 1 may be allocated to the first PV volume, the file read-write subtask 2 may be allocated to the second PV volume, the file read-write subtask 3 may be allocated to the third PV volume, and the file read-write subtask 4 may be allocated to the fourth PV volume.
According to the technical scheme, a plurality of storage volumes are created in the storage pool of the storage end; establishing a plurality of network paths with a storage end according to the number of preset links; and analyzing the SAN mapped storage volumes into block equipment according to the SAN mapping of the storage volumes contained in each network path, wherein the parallelization degree of the host end is the product of the number of the network paths and the number of the storage volumes. In order to reduce the management difficulty of a host end, block devices mapped by the same storage volume on different network paths can be aggregated into a multi-path device according to a multi-path aggregation mechanism; and initializing each multi-path device into a PV volume, and combining all PV volumes into a virtual storage pool, wherein at the moment, the host end only needs to manage and maintain a plurality of PV volumes in the virtual storage pool. Setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, the file read-write task is split according to the width of the strip and then is issued to a target PV volume of the virtual storage pool, and multi-path parallel processing of the file read-write subtasks is achieved according to a network path corresponding to the target PV volume. In the technical scheme, the file reading and writing tasks are sequentially subjected to LVM striping and SAN multipath transmission from the file system of the application end to the physical storage device, and are dispersed to a plurality of PV volumes and a plurality of network paths, so that the parallelization degree of the file system is greatly improved, and the processing efficiency of the file reading and writing tasks is effectively improved.
Fig. 3 is a schematic structural diagram of a task processing device according to an embodiment of the present invention, including a creating unit 31, an establishing unit 32, an analyzing unit 33, an aggregating unit 34, a setting unit 35, and a splitting unit 36;
a creating unit 31, configured to create a plurality of storage volumes in a storage pool on a storage side;
the establishing unit 32 is configured to establish a plurality of network paths with the storage end according to a preset number of links;
a parsing unit 33, configured to parse the SAN-mapped storage volume into the block device according to the SAN mapping of the storage volumes included in each network path
The aggregation unit 34 is configured to map and aggregate the same storage volume on different network paths into a multipath device according to a multipath aggregation mechanism; initializing all multipath equipment into a PV volume; combining all PV volumes into a virtual storage pool;
a setting unit 35, configured to set a stripe width according to a preset size of the service read-write block and the number of PV volumes included in the virtual storage pool;
and the splitting unit 36 is configured to, when receiving the file read-write task, split the file read-write task according to the stripe width and then issue the split file read-write task to the target PV volume in the virtual storage pool, and implement multi-path parallel processing on the file read-write subtasks according to a network path corresponding to the target PV volume.
Optionally, the apparatus includes a setting unit configured to set a capacity of a plurality of storage volumes in the storage side;
and the setting unit is used for setting the capacity of each storage volume according to the total capacity of the file system and the number of the storage volumes contained in the storage end.
Optionally, the splitting unit is specifically configured to split the file read-write task into a plurality of file read-write subtasks according to the stripe width when the file read-write task is received; when the number of the file read-write subtasks is the same as that of the PV volumes contained in the virtual storage pool, a corresponding file read-write subtask is issued to each PV volume; when the number of the file read-write subtasks is smaller than the number of the PV volumes contained in the virtual storage pool, selecting a plurality of target PV volumes with the same number as the file read-write subtasks from the virtual storage pool, and issuing a corresponding file read-write subtask to each target PV volume; and when the number of the file read-write subtasks is larger than that of the PV volumes contained in the virtual storage pool, issuing at least one file read-write subtask to each PV volume.
The description of the features in the embodiment corresponding to fig. 3 may refer to the related description of the embodiment corresponding to fig. 1, and is not repeated here.
According to the technical scheme, a plurality of storage volumes are created in the storage pool of the storage end; establishing a plurality of network paths with a storage end according to the number of preset links; and analyzing the SAN mapped storage volumes into block equipment according to the SAN mapping of the storage volumes contained in each network path, wherein the parallelization degree of the host end is the product of the number of the network paths and the number of the storage volumes. In order to reduce the management difficulty of a host end, block devices mapped by the same storage volume on different network paths can be aggregated into a multi-path device according to a multi-path aggregation mechanism; and initializing each multi-path device into a PV volume, and combining all PV volumes into a virtual storage pool, wherein at the moment, the host end only needs to manage and maintain a plurality of PV volumes in the virtual storage pool. Setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, the file read-write task is split according to the strip width and then is issued to a target PV volume of the virtual storage pool, and multi-path parallel processing of the file read-write sub-tasks is achieved according to a network path corresponding to the target PV volume. In the technical scheme, the file reading and writing tasks are sequentially subjected to LVM striping and SAN multipath transmission from the file system of the application end to the physical storage device, and are dispersed to a plurality of PV volumes and a plurality of network paths, so that the parallelization degree of the file system is greatly improved, and the processing efficiency of the file reading and writing tasks is effectively improved.
Fig. 4 is a schematic structural diagram of a task processing system 40 according to an embodiment of the present invention, which includes a storage side 41 and a host side 42;
a plurality of storage volumes are created in the storage pool of the storage end 41;
a host end 42, configured to establish multiple network paths with the storage end 41 according to a preset number of links; analyzing the SAN mapped storage volumes into block equipment according to SAN mapping of a plurality of storage volumes contained in each network path; according to a multipath aggregation mechanism, aggregating block devices mapped on different network paths by the same storage volume into a multipath device; initializing each multipath device into a PV volume; combining all PV volumes into a virtual storage pool; setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, the file read-write task is split according to the width of the strip and then is issued to a target PV volume of the virtual storage pool, and multi-path parallel processing of the file read-write subtasks is achieved according to a network path corresponding to the target PV volume.
The description of the features in the embodiment corresponding to fig. 4 may refer to the related description of the embodiment corresponding to fig. 1, and is not repeated here.
According to the technical scheme, a plurality of storage volumes are created in the storage pool of the storage end; establishing a plurality of network paths with a storage end according to the number of preset links; and analyzing the SAN mapped storage volumes into block equipment according to SAN mapping of a plurality of storage volumes contained in each network path, wherein the parallelization degree of the host end is the product of the number of the network paths and the number of the storage volumes. In order to reduce the management difficulty of a host end, block devices mapped by the same storage volume on different network paths can be aggregated into a multi-path device according to a multi-path aggregation mechanism; and initializing each multi-path device into a PV volume, and combining all PV volumes into a virtual storage pool, wherein at the moment, the host end only needs to manage and maintain a plurality of PV volumes in the virtual storage pool. Setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, the file read-write task is split according to the width of the strip and then is issued to a target PV volume of the virtual storage pool, and multi-path parallel processing of the file read-write subtasks is achieved according to a network path corresponding to the target PV volume. In the technical scheme, the file reading and writing tasks are sequentially subjected to LVM striping and SAN multipath transmission from the file system of the application end to the physical storage device, and are dispersed to a plurality of PV volumes and a plurality of network paths, so that the parallelization degree of the file system is greatly improved, and the processing efficiency of the file reading and writing tasks is effectively improved.
The task processing method, device and system provided by the embodiment of the invention are described in detail above. The embodiments are described in a progressive mode in the specification, the emphasis of each embodiment is on the difference from the other embodiments, and the same and similar parts among the embodiments can be referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Claims (7)
1. A task processing method, comprising:
creating a plurality of storage volumes in a storage pool of a storage side;
establishing a plurality of network paths with the storage end according to the preset number of links;
analyzing the SAN-mapped storage volumes into block devices according to SAN mapping of the storage volumes contained in each network path;
according to a multipath aggregation mechanism, aggregating block devices mapped on different network paths by the same storage volume into a multipath device; and initializing each of the multi-path devices into a PV volume; combining all the PV volumes into a virtual storage pool;
setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool;
when a file read-write task is received, splitting the file read-write task according to the stripe width and then sending the split file read-write task to a target PV volume of the virtual storage pool, and realizing multi-path parallel processing on the file read-write subtasks according to a network path corresponding to the target PV volume.
2. The task processing method according to claim 1, wherein the method includes, for capacity setting of a plurality of storage volumes in the storage side:
and setting the capacity of each storage volume according to the total capacity of the file system and the number of the storage volumes contained in the storage end.
3. The task processing method according to claim 1 or 2, wherein the splitting the file read-write task according to the stripe width and issuing the split file read-write task to the target PV volume in the virtual storage pool when receiving the file read-write task comprises:
when a file reading and writing task is received, splitting the file reading and writing task into a plurality of file reading and writing subtasks according to the strip width;
when the number of the file read-write subtasks is the same as that of the PV volumes contained in the virtual storage pool, issuing a corresponding file read-write subtask to each PV volume;
when the number of the file read-write subtasks is smaller than the number of the PV volumes contained in the virtual storage pool, selecting a plurality of target PV volumes with the same number as the file read-write subtasks from the virtual storage pool, and issuing a corresponding file read-write subtask to each target PV volume;
and when the number of the file read-write sub-tasks is larger than that of the PV volumes contained in the virtual storage pool, issuing at least one file read-write sub-task to each PV volume.
4. A task processing device is characterized by comprising a creating unit, an establishing unit, an analyzing unit, an aggregating unit, a setting unit and a splitting unit;
the creating unit is used for creating a plurality of storage volumes in the storage pool of the storage end;
the establishing unit is used for establishing a plurality of network paths with the storage end according to the preset number of links;
the analysis unit is configured to analyze the SAN-mapped storage volume into a block device according to the SAN mapping of the plurality of storage volumes included in each network path
The aggregation unit is used for mapping and aggregating the same storage volume on different network paths into a multi-path device according to a multi-path aggregation mechanism; and initializing all the multipath devices to PV volumes; combining all the PV volumes into a virtual storage pool;
the setting unit is used for setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool;
and the splitting unit is used for splitting the file reading and writing task according to the stripe width and then sending the split file reading and writing task to the target PV volume of the virtual storage pool when the file reading and writing task is received, and realizing multi-path parallel processing on the file reading and writing subtasks according to a network path corresponding to the target PV volume.
5. The task processing apparatus according to claim 4, wherein the apparatus includes a setting unit that sets a capacity for a plurality of storage volumes in the storage side;
the setting unit is used for setting the capacity of each storage volume according to the total capacity of the file system and the number of the storage volumes contained in the storage end.
6. The task processing device according to claim 4 or 5, wherein the splitting unit is specifically configured to split the file read-write task into a plurality of file read-write subtasks according to the stripe width when the file read-write task is received; when the number of the file read-write subtasks is the same as that of the PV volumes contained in the virtual storage pool, issuing a corresponding file read-write subtask to each PV volume; when the number of the file read-write subtasks is smaller than the number of the PV volumes contained in the virtual storage pool, selecting a plurality of target PV volumes with the same number as the file read-write subtasks from the virtual storage pool, and issuing a corresponding file read-write subtask to each target PV volume; and when the number of the file read-write subtasks is larger than that of the PV volumes contained in the virtual storage pool, issuing at least one file read-write subtask to each PV volume.
7. A task processing system is characterized by comprising a storage end and a host end;
a plurality of storage volumes are created in the storage pool of the storage end;
the host end is used for establishing a plurality of network paths with the storage end according to the preset number of links; analyzing the SAN-mapped storage volumes into block devices according to SAN mapping of the storage volumes contained in each network path; according to a multipath aggregation mechanism, aggregating block devices mapped on different network paths by the same storage volume into a multipath device; and initializing each of the multi-path devices into a PV volume; combining all the PV volumes into a virtual storage pool; setting the stripe width according to the preset size of the service read-write block and the number of PV volumes contained in the virtual storage pool; when a file read-write task is received, splitting the file read-write task according to the stripe width and then sending the split file read-write task to a target PV volume of the virtual storage pool, and realizing multi-path parallel processing on a file read-write sub-task according to a network path corresponding to the target PV volume.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011231202.9A CN112463030B (en) | 2020-11-06 | 2020-11-06 | Task processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011231202.9A CN112463030B (en) | 2020-11-06 | 2020-11-06 | Task processing method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112463030A CN112463030A (en) | 2021-03-09 |
| CN112463030B true CN112463030B (en) | 2022-08-09 |
Family
ID=74826754
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011231202.9A Active CN112463030B (en) | 2020-11-06 | 2020-11-06 | Task processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463030B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106201357A (en) * | 2016-07-18 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The construction method of a kind of storage pool and system |
| CN106293501A (en) * | 2015-06-25 | 2017-01-04 | 中兴通讯股份有限公司 | Data read-write method and device |
| CN111414228A (en) * | 2020-02-26 | 2020-07-14 | 华为技术有限公司 | Kubernetes-based method for managing storage space and related device |
-
2020
- 2020-11-06 CN CN202011231202.9A patent/CN112463030B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106293501A (en) * | 2015-06-25 | 2017-01-04 | 中兴通讯股份有限公司 | Data read-write method and device |
| CN106201357A (en) * | 2016-07-18 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The construction method of a kind of storage pool and system |
| CN111414228A (en) * | 2020-02-26 | 2020-07-14 | 华为技术有限公司 | Kubernetes-based method for managing storage space and related device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112463030A (en) | 2021-03-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8386610B2 (en) | System and method for automatic storage load balancing in virtual server environments | |
| CN102209087B (en) | Method and system for MapReduce data transmission in data center having SAN | |
| US20140281308A1 (en) | Storage unit selection for virtualized storage units | |
| US20140164612A1 (en) | System and Method for Determining and Visualizing Efficiencies and Risks in Computing Environments | |
| CN109976907B (en) | Task allocation method and system, electronic device and computer readable medium | |
| CN103250143A (en) | Data storage method and storage device | |
| CN103164255B (en) | Virtual machine network communication implementation method and monitor of virtual machine and physical host | |
| CN103608784A (en) | Method for creating network volumes, data storage method, storage device and storage system | |
| CN104216664A (en) | Network volume creating method, data storage method, storage equipment and storage system | |
| CN115834587A (en) | Method and device for selecting target storage server and electronic equipment | |
| CN112463030B (en) | Task processing method, device and system | |
| CN110389814A (en) | A kind of cloud host migration dispatching method, system, terminal and storage medium | |
| US11704034B2 (en) | Load balancing in a storage system | |
| Julián-Moreno et al. | On the feasibility of 40 gbps network data capture and retention with general purpose hardware | |
| US20200073554A1 (en) | Applying Percentile Categories to Storage Volumes to Detect Behavioral Movement | |
| WO2024021470A1 (en) | Cross-region data scheduling method and apparatus, device, and storage medium | |
| CN115729470A (en) | Data writing method and related equipment | |
| CN115640127A (en) | Processor determination method, device, equipment and medium | |
| Ma et al. | MOBBS: A multi-tiered block storage system for virtual machines using object-based storage | |
| CN107220124A (en) | A kind of routing resource and device | |
| CN111930299A (en) | Method for allocating memory units and related device | |
| US11914875B2 (en) | Storage workload allocation based on input/output pattern affinity calculations | |
| WO2018173300A1 (en) | I/o control method and i/o control system | |
| US20220164274A1 (en) | Method, device, and program product for managing storage pool of storage system | |
| US20220244885A1 (en) | Method and system for maximizing performance of a storage system using normalized tokens based on saturation points |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |