CN107301090B - Method and device for setting data processing path for application in storage server - Google Patents
Method and device for setting data processing path for application in storage server Download PDFInfo
- Publication number
- CN107301090B CN107301090B CN201610232333.6A CN201610232333A CN107301090B CN 107301090 B CN107301090 B CN 107301090B CN 201610232333 A CN201610232333 A CN 201610232333A CN 107301090 B CN107301090 B CN 107301090B
- Authority
- CN
- China
- Prior art keywords
- data processing
- parameter value
- processing path
- application
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Abstract
The invention discloses a method and a device for setting a data processing path for an application in a storage server, and belongs to the field of data storage. The method comprises the following steps: receiving a target parameter value of a target data processing path set for an application and sent by an application server; determining a parameter value corresponding to each data processing path in the storage server and the target parameter value; setting the determined data processing path corresponding to the parameter value closest to the target parameter value as a target data processing path of the application. By acquiring target parameter values reflecting requirements of application performance, cost and the like and setting the data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the target data processing path can meet the requirements of the application performance, cost and the like, and the data I/O performance of the application is improved.
Description
Technical Field
The present invention relates to the field of data storage, and in particular, to a method and an apparatus for setting a data processing path for an application in a storage server.
Background
With the popularization of the internet and the development of social informatization, people's lives and works have been kept away from various applications, such as database applications, multimedia applications, virtual desktop applications, and the like. The storage system may provide a data storage service for a plurality of applications, and the storage system includes a storage server, and the storage server may set a corresponding data processing path for the application based on various software and hardware resources included in the storage system, so that the application may be performed according to the set data processing path when performing data I/O (Input Output).
In the prior art, a storage server may randomly set a data processing path for an application based on existing software and hardware resources. However, although the existing random allocation mechanism can set a data processing path for an application so that the application can complete basic data processing, different applications have different requirements on performance, cost, and the like of data processing, for example, a virtual desktop application has a higher requirement on delay of data processing, a multimedia application has a lower requirement on delay and has a higher requirement on cost, and the like. The data processing path set based on the random allocation mechanism cannot meet the requirements of application performance, cost and the like, so that the data I/O performance of the application is affected.
Disclosure of Invention
In order to solve the problems in the prior art, the application provides a method and a device for setting a data processing path for an application in a storage server. The technical scheme is as follows:
in a first aspect, a method for setting a data processing path for an application in a storage server is provided, where the storage server is provided with a plurality of data processing paths, and the data processing paths are a plurality of tasks that the storage server needs to execute to complete data I/O of the application; the method comprises the following steps:
receiving a target parameter value of a target data processing path set for an application and sent by an application server; determining a parameter value corresponding to each data processing path in the storage server and the target parameter value; setting the determined data processing path corresponding to the parameter value closest to the target parameter value as a target data processing path of the application. By acquiring target parameter values reflecting requirements of application performance, cost and the like and setting the data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the target data processing path can meet the requirements of the application performance, cost and the like, and the data I/O performance of the application is improved.
With reference to the first aspect, in a first possible implementation manner of the first aspect, each task in the storage server includes at least one processing manner, and one processing manner is selected from the at least one processing manner of each task to form the plurality of data processing paths. By providing multiple processing modes for each task, the diversity of data processing path selection is increased, and the storage server can better set a data processing path meeting the requirements for the application.
With reference to the first aspect, in a second possible implementation manner of the first aspect, the determining a parameter value corresponding to the target parameter for each data processing path in the storage server includes: acquiring a plurality of tasks corresponding to each data processing path and at least one processing mode of each task; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the plurality of tasks and at least one processing mode of each task. By calculating the parameter value corresponding to each data processing path and the target parameter value, the data processing path meeting the requirement can be set for the application more accurately.
With reference to the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the calculating, according to the multiple tasks and at least one processing manner of each task, a parameter value of each data processing path corresponding to the target parameter value includes: generating a Directed Acyclic Graph (DAG) according to the plurality of tasks and at least one processing mode of each task, wherein the DAG is used for indicating a plurality of data processing paths of the application; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the DAG. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
With reference to the first aspect, in a fourth possible implementation manner of the first aspect, the setting, as the target data processing path of the application, the data processing path corresponding to the determined parameter value closest to the target parameter value includes: judging whether the parameter value corresponding to the target parameter value of each data processing path is not greater than the target parameter value; if a parameter value corresponding to the target parameter value of one data processing path is not larger than the target parameter value, setting the data processing path as a target data processing path of the application; if the parameter value corresponding to the target parameter value of at least two data processing paths is not larger than the target parameter value, acquiring a first data processing path with the minimum parameter value from the at least two data processing paths, and setting the first data processing path as the target data processing path of the application. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
With reference to the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, after the determining whether the parameter value corresponding to the target parameter value of each data processing path is not greater than the target parameter value, the method further includes: if the parameter value corresponding to the target parameter value of each data processing path is greater than the target parameter value, acquiring a second data processing path with the minimum parameter value from the plurality of applied data processing paths; setting the second data processing path as a target data processing path of the application. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
With reference to the first aspect, in a sixth possible implementation manner of the first aspect, after the setting the determined data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the method further includes: and when a data writing request of the application is received, storing the data of the application according to the target data processing path. The data I/O performance of the application is improved by storing the data of the application according to the target data processing path meeting the requirements of application performance, cost and the like.
In a second aspect, there is provided an apparatus for setting a data processing path for an application in a storage server, the apparatus comprising a plurality of functional modules for executing the method according to the first aspect. In one possible implementation manner, the apparatus further includes other functional modules for executing the method described in the foregoing various possible implementation manners of the first aspect. By acquiring target parameter values reflecting requirements of application performance, cost and the like and setting the data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the target data processing path can meet the requirements of the application performance, cost and the like, and the data I/O performance of the application is improved.
In a third aspect, a server is provided that includes a processor, a memory to store processor-executable instructions; wherein the processor is configured to:
receiving a target parameter value of a target data processing path set for an application and sent by an application server; determining a parameter value corresponding to each data processing path in the storage server and the target parameter value; setting the determined data processing path corresponding to the parameter value closest to the target parameter value as a target data processing path of the application. By acquiring target parameter values reflecting requirements of application performance, cost and the like and setting the data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the target data processing path can meet the requirements of the application performance, cost and the like, and the data I/O performance of the application is improved.
With reference to the third aspect, in a first possible implementation manner of the third aspect, each task in the storage server includes at least one processing manner, and one processing manner is selected from the at least one processing manner of each task to form the plurality of data processing paths. By providing multiple processing modes for each task, the diversity of data processing path selection is increased, and the storage server can better set a data processing path meeting the requirements for the application.
With reference to the third aspect, in a second possible implementation manner of the third aspect, the determining a parameter value corresponding to the target parameter for each data processing path in the storage server includes: acquiring a plurality of tasks corresponding to each data processing path and at least one processing mode of each task; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the plurality of tasks and at least one processing mode of each task. By calculating the parameter value corresponding to each data processing path and the target parameter value, the data processing path meeting the requirement can be set for the application more accurately.
With reference to the second possible implementation manner of the third aspect, in a third possible implementation manner of the third aspect, the processor is configured to: generating a Directed Acyclic Graph (DAG) according to the plurality of tasks and at least one processing mode of each task, wherein the DAG is used for indicating a plurality of data processing paths of the application; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the DAG. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
With reference to the third aspect, in a fourth possible implementation manner of the third aspect, the processor is configured to: judging whether the parameter value corresponding to the target parameter value of each data processing path is not greater than the target parameter value; if a parameter value corresponding to the target parameter value of one data processing path is not larger than the target parameter value, setting the data processing path as a target data processing path of the application; if the parameter value corresponding to the target parameter value of at least two data processing paths is not larger than the target parameter value, acquiring a first data processing path with the minimum parameter value from the at least two data processing paths, and setting the first data processing path as the target data processing path of the application. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
With reference to the fourth possible implementation manner of the third aspect, in a fifth possible implementation manner of the third aspect, the processor is configured to: if the parameter value corresponding to the target parameter value of each data processing path is greater than the target parameter value, acquiring a second data processing path with the minimum parameter value from the plurality of applied data processing paths; setting the second data processing path as a target data processing path of the application. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
With reference to the third aspect, in a sixth possible implementation manner of the third aspect, the processor is configured to: and when a data writing request of the application is received, storing the data of the application according to the target data processing path. The data I/O performance of the application is improved by storing the data of the application according to the target data processing path meeting the requirements of application performance, cost and the like.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a storage system according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for setting a data processing path for an application in a storage server according to an embodiment of the present invention;
FIG. 3 is a DAG diagram of a data processing path provided by an embodiment of the invention;
fig. 4 is a block diagram of an apparatus for setting a data processing path for an application in a storage server according to an embodiment of the present invention;
fig. 5 is a block diagram of an apparatus for setting a data processing path for an application in a storage server according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
Fig. 1 is a schematic structural diagram of a storage system according to an embodiment of the present invention. Referring to fig. 1, a storage server and a storage medium are included. In addition, the storage system is further connected to at least one application server through a network, only one application server is shown in fig. 1, and the number of the application servers is not particularly limited in the embodiment of the present invention.
Each application server in the at least one application server includes hardware resources such as a Central Processing Unit (CPU), a memory, a cache, and software modules such as an application and an operating system, and is configured to obtain a target parameter value of a target data processing path of the application, and send the target parameter value to the storage server through a designated interface, so that the storage server can set, in the multiple data processing paths of the application, a data processing path that satisfies the target parameter value for the application according to the target parameter value of the application. An application server may provide services for one or more applications, where the applications include database applications, Virtual Desktop Infrastructure (VDI) applications, multimedia applications, and the like; the operating system includes a file system, volume management, multipath software, etc. The database application may include an Online Transaction Processing (OLTP) type database application, and an Online analytical Processing (OLAP) type database application.
The storage medium includes a Solid State Drive (SSD) and a Hard Disk Drive (HDD) for storing application data. The storage server includes hardware resources such as a Central Processing Unit (CPU), a memory, and a cache. The storage server is used for providing data storage service for the application in the application server. The storage server is provided with a plurality of data processing paths, and the data processing paths are a plurality of tasks which need to be executed by the storage server to complete data I/O of application, such as a storage medium type selection task, a data caching task, a data deduplication task, a data compression task, a data encoding task, a data redundancy task, a data addressing task and the like. The storage server is used for receiving the target parameter value of the application sent by the application server, and selecting a data processing path meeting the target parameter value from a plurality of data processing paths of the application to be set as the data processing path of the application, so as to provide access to data in the storage server through the set data processing path, such as data reading, writing, deleting and the like.
Fig. 2 is a flowchart of a method for setting a data processing path for an application in a storage server according to an embodiment of the present invention, and with reference to fig. 2, the method includes the following steps:
201. and the storage server receives a target parameter value of a target data processing path set for the application, which is sent by the application server.
It should be noted that, when performing data I/O, the application server generally needs to perform multiple tasks, such as a storage medium type selection task, a data caching task, a data deduplication task, a data compression task, a data encoding task, a data redundancy task, a data addressing task, and the like. The physical environment of the storage system is fixed and unchangeable for a certain time, wherein the physical environment includes storage space, type and size of storage medium, and the like. The storage server is provided with a plurality of data processing paths, and the data processing paths are a plurality of tasks which are required to be executed by the storage server to complete data I/O of the application.
For a plurality of tasks, the storage server provides at least one processing mode for each task, that is, each task includes at least one processing mode, for example, for a data compression task, the storage server provides two data compression modes for the data compression task: data compression scheme 1 and data compression scheme 2. And selecting one processing mode from the at least one processing mode of each task to form a plurality of data processing paths.
In the data I/O process, each processing mode corresponds to multiple processing consumptions, such as cost consumption, delay consumption, data reliability consumption, and the like. The embodiment of the present invention refers to the multiple processing consumptions corresponding to each processing manner as the parameter values corresponding to each processing manner, and the parameter values may include multiple parameter values, such as a cost parameter value, a delay parameter value, a data reliability parameter value, and the like. Accordingly, the parameter value of each processing manner is used to indicate a plurality of processing consumptions corresponding to each processing manner, for example, a plurality of processing consumptions such as a cost consumption corresponding to the cost parameter value, a delay consumption corresponding to the delay parameter value, a data reliability consumption corresponding to the data reliability parameter value, and the like.
It should be noted that each data processing path is formed by selecting one processing manner from at least one processing manner of each task, that is, each data processing path includes a plurality of processing manners. Correspondingly, the parameter value of the data I/O corresponding to each data processing path is the sum of each parameter value corresponding to a plurality of data processing modes in each data processing path.
The target data processing path refers to a data processing path set by the storage server for the application, so that when the application stores data, the application processes the data according to a processing mode included in the target data processing path.
In the embodiment of the invention, an application administrator can preset a target parameter value of a target data processing path for an application through an application server, wherein the target parameter value is used for indicating each processing consumption expected when the application performs data I/O.
Specifically, the application server may send the target parameter value to the storage server through the designated interface, and the storage server selects, from the plurality of data processing paths, a data processing path that satisfies the target parameter value as the target data processing path according to the target parameter value. Wherein the target parameter values include a target cost, a target latency, and a target data reliability. The target cost is used to indicate cost consumption of the data processing path, and may include a target storage cost of the application data, such as a storage medium, a storage space, and the like for storing the application data; the target time delay is used for indicating the time delay consumption of the data processing path and is a target time interval from the time when the application sends a data I/O request to the storage equipment to the time when a data I/O response is received; the target data reliability is used to indicate data reliability consumption of the data processing path, that is, a requirement for integrity or consistency of data requested by the data I/O request is applied, for example, the requirement for integrity of the data requested by the data I/O request is 95%, that is, the data I/O request tolerates 5% of data loss at maximum, and the target data reliability consumption may be 5%.
It should be noted that, the target parameter values of the application may include parameter values of a plurality of target parameters, and when the target parameter values are set, one or more of the target parameter values may be set. The user may refer to an application developer or an application administrator, etc.
In the embodiment of the invention, the application server can provide a configuration interface of the target parameter value of the application for the user, and in the configuration interface of the target parameter value, the user can set the target cost, the target time delay and the target data reliability of the application. The configuration interface of the target parameter value may be displayed during the process of installing the application by the user, or displayed after the user triggers the configuration operation of the application, or embedded in software that is specially used for configuring the processing consumption target of the application, which is not specifically limited in the embodiment of the present invention.
202. And the storage server determines the parameter value corresponding to the target parameter value of each data processing path in the storage server.
In the embodiment of the present invention, after determining the target parameter values of the target data processing paths to be applied, the parameter values corresponding to the target parameter values of each data processing path, that is, the parameter values corresponding to the target parameter values of the plurality of target data processing paths, may be obtained correspondingly according to the plurality of target parameters corresponding to the target parameter values. Because each task of the application corresponds to a plurality of processing modes, different processing modes are adopted, and the applied parameter values are different, so that the parameter values corresponding to different data processing paths are different.
The process of determining the parameter value corresponding to the target parameter value in each data processing path in the storage server may be: acquiring a plurality of tasks corresponding to each data processing path and at least one processing mode of each task; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the plurality of tasks and at least one processing mode of each task.
It should be noted that, during data I/O, the execution order of the multiple tasks of the application is different, so that the priority level of each task is different, for example, if the compression task is to be executed preferentially to the cache task, the priority level of the compression task is higher than that of the cache task. After one processing mode is selected from the at least one processing mode of each task, the priority corresponding to the task may be considered as the priority of the selected processing mode.
Specifically, in order to calculate the parameter value of each data processing path more easily and quickly, in the embodiment of the present invention, the storage server may perform DAG (Directed Acyclic Graph) modeling according to the multiple data processing paths of the application, so that each path in the DAG indicates one data processing path. Then, calculating a parameter value corresponding to the target parameter value of each data processing path of the application according to the established model, wherein the process may be as follows: generating a Directed Acyclic Graph (DAG) according to the plurality of tasks and at least one processing mode of each task, wherein the DAG is used for indicating a plurality of data processing paths of the application; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the DAG.
Specifically, the DAG generation process may be: acquiring the priority of each task, and determining each processing mode corresponding to each task as a node of the DAG; determining the level of each processing mode corresponding node corresponding to each task in the DAG according to the priority of each task, wherein the higher the priority of the task is, the higher the level of the processing mode corresponding node is; determining an initial node and an end node of the DAG, the initial node being a start point of each data processing path and being highest at a level in the DAG, the end node being an end point of each data processing path and being lowest at the level in the DAG; adding directed edges between nodes of adjacent levels, wherein the directed edges point to nodes of lower levels in the adjacent levels, namely, each directed edge connects two processing modes with different priorities; the weight of the directed edge can be set as the processing consumption of the processing mode pointed by the directed edge, and finally a directed acyclic graph taking the processing mode as a node and the parameter value of the processing mode as the weight of the directed edge is formed. And when the target parameter value comprises parameter values corresponding to a plurality of parameters, the parameter value of the processing mode also comprises the parameter values corresponding to the plurality of parameters. Correspondingly, each directed edge weight in the DAG is a parameter value corresponding to the multiple parameters of the processing mode.
It should be noted that, the initial node of the DAG corresponds to a starting point of data I/O when the application performs data storage, for example, a time when the data I/O enters the storage system from the application layer (i.e., a time when the storage system receives the data I/O of the application); the end point of the DAG corresponds to the end point of data I/O when the application performs data storage, such as when the data I/O returns from the storage system to the application layer (i.e., the time when the application receives the data I/O returned by the storage system).
The following exemplifies the DAG generation process in conjunction with the above DAG generation method. For example, the application includes 4 tasks, the 4 tasks correspond to 4 priorities, the task with the first priority is a data compression task, and the processing mode corresponding to the data compression task includes: a compression algorithm A, a compression algorithm B, no compression, a deduplication algorithm A, a deduplication algorithm B and no deduplication; the task of the second priority is a data caching task, and the processing mode corresponding to the data caching task comprises the following steps: SSD cache, NVRAM (Non-Volatile Random Access Memory) cache, Memory cache, no cache; the task of the third priority is a data redundancy task, and the processing mode corresponding to the data redundancy task comprises the following steps: DHT (DistributedHash Table), EC (Erasure Code), triple copy, no redundancy; the task with the fourth priority is a data backup task, and the processing mode corresponding to the data backup task comprises the following steps: dual live, global backup, block service, file service, object service.
If the initial node of the DAG is P0, the end node is Pe. P0 is highest and Pe is lowest in the DAG. In particular implementations, the initial node may be set to a level of 0 by default. According to different priorities of the tasks, a first level in the DAG corresponding to the first priority can be obtained, a second level in the DAG corresponding to the second priority is obtained, a third level in the DAG corresponding to the third level is obtained, a fourth level in the DAG corresponding to the fourth level is obtained, and the level of the end node is 5. For convenience of description, nodes corresponding to multiple processing modes corresponding to the tasks with 4 priorities in the DAG are numbered, and the specific correspondence is as follows: compression algorithm a (P11), compression algorithm B (P12), no compression C (P13), deduplication algorithm a (P14), deduplication algorithm B (P15), no deduplication (P16); SSD cache (P21), NVRAM cache (P22), memory cache (P23), no cache (P24); DHT (P31), EC (P32), triple copy (P33), no redundancy (P34), live-live (P41), global backup (P42), block service (P43), file service (P44) and object service (P45). Wherein, the content in the "()" is the node number corresponding to the processing mode.
According to the levels in the DAG in the processing mode, directed edges pointing to nodes of lower levels are added between nodes of two adjacent levels, and a DAG graph for constructing the processing mode shown in fig. 3 can be obtained. In FIG. 3, the DAG shows a plurality of data processing paths for the application, where each path contained in the DAG corresponds to each data processing path for the application.
It should be noted that, the priorities of the different tasks are only used for illustration, and the embodiment of the present invention is not limited to this specifically.
In the embodiment of the invention, because different applications comprise different tasks and the processing modes corresponding to the different tasks are different, DAGs for the different applications can be generated according to the processing consumption targets of the different applications. When the processing mode in the storage server is changed, for example, a new processing mode is added to a task included in the application, the DAG of the application can be automatically regenerated according to the changed processing mode and the parameter value of each changed processing mode, and the data processing path meeting the target parameter value of the application is obtained again. Or after a new processing mode is added, prompting a user whether to reset the applied target parameter value, namely resetting the target cost, the target time delay and the target data reliability; and if the target parameter value of the application is reset by the user, regenerating the DAG of the application according to the changed processing modes and the parameter value of each changed processing mode based on the target parameter value reset by the user, and acquiring the data processing path meeting the new target parameter value again. The DAG of the application is automatically regenerated according to the changed processing mode, and the data processing path meeting the target parameter value is obtained again, so that the flexibility of setting the data processing path is improved, and the efficiency of setting the data processing path is improved.
In this embodiment of the present invention, after the DAG is generated, the parameter value of each data processing path may be calculated as follows:
and acquiring all paths from the initial node to the end node, and summing the weights of the directed edges contained in each path to obtain the parameter value of each data processing path.
203. And the storage server sets the determined data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application.
In this embodiment of the present invention, the process of setting, in the plurality of data processing paths, the data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application may be: firstly, judging whether the parameter value corresponding to the target parameter value of each data processing path is not more than the target parameter value. Secondly, there are three possible situations according to the difference of the judgment results:
first, if a data processing path exists, and a parameter value corresponding to a target parameter value is not greater than the target parameter value, setting the data processing path as a target data processing path of an application.
And secondly, if the parameter values corresponding to the target parameter values of at least two data processing paths are not larger than the target parameter values, acquiring a first data processing path with the minimum parameter value from the at least two data processing paths, and setting the first data processing path as the target data processing path of the application.
Thirdly, if the parameter value corresponding to the target parameter value of each data processing path is larger than the target parameter value, acquiring a second data processing path with the minimum parameter value from the applied multiple data processing paths; the second data processing path is set as a target data processing path for the application.
It should be noted that, in the third case, a minimum parameter value threshold may be set, and if the calculated parameter value of the second data processing path is still greater than the minimum parameter value threshold, the current storage system cannot meet the target parameter value set by the user, and a prompt message for resetting the target parameter value may be displayed to the user.
In the embodiment of the invention, the problem of acquiring the data processing path corresponding to the parameter value closest to the target parameter value through the DAG model is converted into the problem of acquiring the shortest path in the DAG.
The basic principle of calculating the shortest path in the DAG is to adopt a dynamic programming method: v denotes a set of all nodes in the constructed DAG, S denotes a set of nodes for which the shortest path has been found, and V-S ═ T denotes a set of nodes for which the shortest path has not been determined. Initially, S is { V0}, V0 is the starting point, T is { the rest nodes }, and for any Vi (Vi belongs to T), if there is a direct connection from V0 to Vi < V0, Vi >, < V0, Vi > is the weight on the arc, and if not, < V0, Vi > is set to infinity. Selecting a node W with the minimum distance V0 from T, adding S, and modifying the distance values of other nodes in T: if W is added as an intermediate node and the distance value from V0 to Vi is shorter than the path without W, then the distance value is modified. The above steps are repeated until all nodes are included in S, i.e., S ═ V.
204. And when receiving a data writing request of the application, the storage server stores the data of the application according to the target data processing path.
In the embodiment of the present invention, after the target data processing path of the application is determined, the plurality of tasks corresponding to the target data processing path, the processing manner of each task, and the parameter value of the target data processing path may be output, so that a user may clearly know the data processing path of the application and adopt the parameter value of the data processing path, that is, a plurality of processing consumptions, such as cost consumption, delay consumption, data reliability consumption, and the like, corresponding to the data processing path. And if the user determines that the output target data processing path is taken as the final data processing path of the application, storing the data of the application according to the determined target data processing path when the storage server receives a data writing request of the application.
It should be noted that, when the application interacts with the storage system, the application includes not only the write operation, but also operations such as read and delete, which is not specifically limited in this embodiment of the present invention. Aiming at different operations corresponding to different data I/O (input/output) of the application, the storage server receives corresponding operation requests of the application, executes data reading, deleting and other operations through the storage device according to the corresponding operation requests, and returns processing results to the application.
It should be noted that the method provided in the embodiment of the present invention may be applied to a high-end storage array, a mass storage cloud storage system, a Doradoa V5 full flash memory array storage system, and the like, and this is not specifically limited in the embodiment of the present invention.
According to the method provided by the embodiment of the invention, the target parameter value of the target data processing path is obtained, and the data processing path corresponding to the parameter value closest to the target parameter value is set as the target data processing path of the application in the plurality of applied data processing paths, so that the application data is processed in an optimal processing mode under the constraint factors of resources, performance, cost and the like. Furthermore, the problem of obtaining the optimal data processing path is converted into the problem of obtaining the shortest path in the DAG model, so that the calculation amount of data processing is greatly simplified, and the process of obtaining the optimal data processing path is simpler, more convenient and faster.
Fig. 4 is a block diagram of an apparatus for setting a data processing path for an application in a storage server according to an example of the present invention, including: a receiving module 401, an obtaining module 402, and a setting module 403.
The receiving module 401 is configured to receive a target parameter value of a target data processing path set for an application, where the target parameter value is sent by an application server;
an obtaining module 402, configured to determine a parameter value corresponding to each data processing path in the storage server and the target parameter value;
a setting module 403, configured to set the determined data processing path corresponding to the parameter value closest to the target parameter value as a target data processing path of the application.
Optionally, each task in the storage server includes at least one processing method, and one processing method is selected from the at least one processing method of each task to form a plurality of data processing paths. By providing multiple processing modes for each task, the diversity of data processing path selection is increased, and the storage server can better set a data processing path meeting the requirements for the application.
Optionally, the obtaining module is configured to obtain a plurality of tasks corresponding to each data processing path and at least one processing manner of each task; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the plurality of tasks and at least one processing mode of each task. By calculating the parameter value corresponding to each data processing path and the target parameter value, the data processing path meeting the requirement can be set for the application more accurately.
Optionally, the obtaining module is configured to generate a directed acyclic graph DAG according to the plurality of tasks and at least one processing manner of each task, where the DAG is configured to indicate a plurality of data processing paths of the application; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the DAG. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
Optionally, the setting module is configured to determine whether a parameter value corresponding to the target parameter value is not greater than the target parameter value for each data processing path; if the parameter value corresponding to the target parameter value of one data processing path is not larger than the target parameter value, setting the data processing path as an applied target data processing path; if the parameter values corresponding to the target parameter values of the at least two data processing paths are not larger than the target parameter values, acquiring a first data processing path with the minimum parameter value from the at least two data processing paths, and setting the first data processing path as an applied target data processing path. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
Optionally, the setting module is configured to, if the parameter value corresponding to the target parameter value is greater than the target parameter value in each of the data processing paths, obtain a second data processing path with a minimum parameter value in the applied multiple data processing paths; the second data processing path is set as a target data processing path for the application. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
Optionally, the apparatus further comprises:
and the storage module is used for storing the applied data according to the target data processing path when receiving the data writing request of the application. The data I/O performance of the application is improved by storing the data of the application according to the target data processing path meeting the requirements of application performance, cost and the like.
According to the device provided by the embodiment of the invention, the target parameter values reflecting the requirements of the application such as performance and cost are obtained, and the data processing path corresponding to the parameter value closest to the target parameter value is set as the target data processing path of the application, so that the target data processing path can meet the requirements of the application such as performance and cost, and the data I/O performance of the application is improved.
It should be noted that: in the data processing path obtaining apparatus in the storage system provided in the foregoing embodiment, when obtaining the data processing path, only the division of each functional module is illustrated, and in practical applications, the function setting may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the fault-tolerant processing apparatus and the fault-tolerant processing method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments in detail and are not described herein again.
Fig. 5 is a block diagram of an apparatus for setting a data processing path for an application in a storage server according to an embodiment of the present invention, for example, the apparatus 500 may be provided as a server. Referring to fig. 5, the apparatus 500 includes a processing component 522 that further includes one or more processors and memory resources, represented by memory 532, for storing instructions, such as applications, that are executable by the processing component 522. The application programs stored in memory 532 may include one or more modules that each correspond to a set of instructions. Further, the processing component 522 is configured to execute instructions to perform the data processing path acquisition method in the storage system described above.
The apparatus 500 may also include a power component 526 configured to perform power management of the apparatus 500, a wired or wireless network interface 550 configured to connect the apparatus 500 to a network, and an input/output (I/O) interface 558. The apparatus 500 may operate based on an operating system, such as Windows Server, stored in the memory 532TM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTMOr the like.
The processing component 522 is configured to execute instructions to execute the data processing path obtaining method in the storage system, and includes the following steps:
receiving a target parameter value of a target data processing path set for an application and sent by an application server; determining a parameter value corresponding to each data processing path in the storage server and the target parameter value; setting the determined data processing path corresponding to the parameter value closest to the target parameter value as a target data processing path of the application. By acquiring target parameter values reflecting requirements of application performance, cost and the like and setting the data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the target data processing path can meet the requirements of the application performance, cost and the like, and the data I/O performance of the application is improved.
Optionally, each task in the storage server includes at least one processing method, and one processing method is selected from the at least one processing method of each task to form the plurality of data processing paths. By providing multiple processing modes for each task, the diversity of data processing path selection is increased, and the storage server can better set a data processing path meeting the requirements for the application.
Optionally, the determining a parameter value corresponding to the target parameter for each data processing path in the storage server includes: acquiring a plurality of tasks corresponding to each data processing path and at least one processing mode of each task; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the plurality of tasks and at least one processing mode of each task. By calculating the parameter value corresponding to each data processing path and the target parameter value, the data processing path meeting the requirement can be set for the application more accurately.
Optionally, the calculating, according to the multiple tasks and at least one processing manner of each task, a parameter value corresponding to the target parameter value for each data processing path includes: generating a Directed Acyclic Graph (DAG) according to the plurality of tasks and at least one processing mode of each task, wherein the DAG is used for indicating a plurality of data processing paths of the application; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the DAG. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
Optionally, the setting the determined data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application includes: judging whether the parameter value corresponding to the target parameter value of each data processing path is not greater than the target parameter value; if a parameter value corresponding to the target parameter value of one data processing path is not larger than the target parameter value, setting the data processing path as a target data processing path of the application; if the parameter value corresponding to the target parameter value of at least two data processing paths is not larger than the target parameter value, acquiring a first data processing path with the minimum parameter value from the at least two data processing paths, and setting the first data processing path as the target data processing path of the application. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
Optionally, after determining whether the parameter value corresponding to the target parameter value of each data processing path is not greater than the target parameter value, the method further includes: if the parameter value corresponding to the target parameter value of each data processing path is greater than the target parameter value, acquiring a second data processing path with the minimum parameter value from the plurality of applied data processing paths; setting the second data processing path as a target data processing path of the application. By converting the problem of acquiring the optimal data processing path into the problem of acquiring the shortest path in the DAG model, the calculation amount of data processing is greatly simplified, and the acquisition process of the optimal data processing path is simpler, more convenient and faster.
Optionally, after setting the determined data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the method further includes: and when a data writing request of the application is received, storing the data of the application according to the target data processing path. The data I/O performance of the application is improved by storing the data of the application according to the target data processing path meeting the requirements of application performance, cost and the like.
According to the device provided by the embodiment of the invention, the target parameter values reflecting the requirements of the application such as performance and cost are obtained, and the data processing path corresponding to the parameter value closest to the target parameter value is set as the target data processing path of the application, so that the target data processing path can meet the requirements of the application such as performance and cost, and the data I/O performance of the application is improved.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (14)
1. A method for setting a data processing path for an application in a storage server, wherein a plurality of data processing paths are set for the application in the storage server, and the data processing paths are a plurality of tasks required to be executed by the storage server to complete data I/O of the application;
characterized in that the method comprises:
receiving a target parameter value of a target data processing path set for an application and sent by an application server;
determining a parameter value corresponding to each data processing path in the storage server and the target parameter value;
setting the determined data processing path corresponding to the parameter value closest to the target parameter value as a target data processing path of the application.
2. The method of claim 1, wherein each task in the storage server includes at least one processing mode, and wherein selecting one processing mode from the at least one processing mode for each task forms the plurality of data processing paths.
3. The method of claim 1, wherein the determining a parameter value corresponding to the target parameter for each data processing path in the storage server comprises:
acquiring a plurality of tasks corresponding to each data processing path and at least one processing mode of each task;
and calculating the parameter value corresponding to the target parameter value of each data processing path according to the plurality of tasks and at least one processing mode of each task.
4. The method of claim 3, wherein the calculating the parameter value corresponding to the target parameter value for each data processing path according to the plurality of tasks and at least one processing manner of each task comprises:
generating a Directed Acyclic Graph (DAG) according to the plurality of tasks and at least one processing mode of each task, wherein the DAG is used for indicating a plurality of data processing paths of the application;
and calculating the parameter value corresponding to the target parameter value of each data processing path according to the DAG.
5. The method of claim 1, wherein setting the determined data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application comprises:
judging whether the parameter value corresponding to the target parameter value of each data processing path is not greater than the target parameter value;
if a parameter value corresponding to the target parameter value of one data processing path is not larger than the target parameter value, setting the data processing path as a target data processing path of the application;
if the parameter value corresponding to the target parameter value of at least two data processing paths is not larger than the target parameter value, acquiring a first data processing path with the minimum parameter value from the at least two data processing paths, and setting the first data processing path as the target data processing path of the application.
6. The method of claim 5, wherein after determining whether the parameter value corresponding to the target parameter value for each data processing path is not greater than the target parameter value, the method further comprises:
if the parameter value corresponding to the target parameter value of each data processing path is greater than the target parameter value, acquiring a second data processing path with the minimum parameter value from the plurality of applied data processing paths;
setting the second data processing path as a target data processing path of the application.
7. The method of claim 1, wherein after setting the determined data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application, the method further comprises:
and when a data writing request of the application is received, storing the data of the application according to the target data processing path.
8. The device for setting a data processing path for an application in a storage server, wherein a plurality of data processing paths are set for the application in the storage server, and the data processing paths are a plurality of tasks required to be executed by the storage server to complete data I/O of the application;
characterized in that the device comprises:
the receiving module is used for receiving a target parameter value of a target data processing path which is sent by the application server and is set for the application;
the acquisition module is used for determining the parameter value corresponding to the target parameter value of each data processing path in the storage server;
and the setting module is used for setting the determined data processing path corresponding to the parameter value closest to the target parameter value as the target data processing path of the application.
9. The apparatus of claim 8, wherein each task in the storage server comprises at least one processing mode, and wherein one processing mode selected from the at least one processing mode of each task forms the plurality of data processing paths.
10. The apparatus according to claim 8, wherein the obtaining module is configured to obtain a plurality of tasks corresponding to each data processing path and at least one processing manner of each task; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the plurality of tasks and at least one processing mode of each task.
11. The apparatus of claim 10, wherein the obtaining module is configured to generate a Directed Acyclic Graph (DAG) indicating a plurality of data processing paths of the application according to the plurality of tasks and at least one processing manner of each task; and calculating the parameter value corresponding to the target parameter value of each data processing path according to the DAG.
12. The apparatus according to claim 8, wherein the setting module is configured to determine whether a parameter value corresponding to the target parameter value of each data processing path is not greater than the target parameter value; if a parameter value corresponding to the target parameter value of one data processing path is not larger than the target parameter value, setting the data processing path as a target data processing path of the application; if the parameter value corresponding to the target parameter value of at least two data processing paths is not larger than the target parameter value, acquiring a first data processing path with the minimum parameter value from the at least two data processing paths, and setting the first data processing path as the target data processing path of the application.
13. The apparatus according to claim 12, wherein the setting module is configured to, if the parameter value corresponding to the target parameter value in each of the data processing paths is greater than the target parameter value, obtain a second data processing path with a smallest parameter value in the plurality of data processing paths of the application; setting the second data processing path as a target data processing path of the application.
14. The apparatus of claim 8, further comprising:
and the storage module is used for storing the data of the application according to the target data processing path when receiving the data writing request of the application.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610232333.6A CN107301090B (en) | 2016-04-14 | 2016-04-14 | Method and device for setting data processing path for application in storage server |
| PCT/CN2017/080480 WO2017177953A1 (en) | 2016-04-14 | 2017-04-13 | Method and apparatus for setting data processing path for application in storage server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610232333.6A CN107301090B (en) | 2016-04-14 | 2016-04-14 | Method and device for setting data processing path for application in storage server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107301090A CN107301090A (en) | 2017-10-27 |
| CN107301090B true CN107301090B (en) | 2020-11-03 |
Family
ID=60042364
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610232333.6A Active CN107301090B (en) | 2016-04-14 | 2016-04-14 | Method and device for setting data processing path for application in storage server |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107301090B (en) |
| WO (1) | WO2017177953A1 (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110826057B (en) * | 2018-08-09 | 2023-06-02 | 阿里巴巴集团控股有限公司 | Analysis method for data processing path, computer device, and storage medium |
| CN111399753B (en) * | 2019-08-23 | 2022-08-05 | 杭州海康威视系统技术有限公司 | Method and device for writing pictures |
| CN111061711B (en) * | 2019-11-28 | 2023-09-01 | 同济大学 | Big data stream unloading method and device based on data processing behavior |
| CN111428910B (en) * | 2020-02-28 | 2023-12-05 | 西安电子科技大学 | Vehicle shortest time traffic path processing method based on delay Petri network |
| CN116225346B (en) * | 2023-05-09 | 2023-07-25 | 此芯科技(上海)有限公司 | Memory data access method and electronic equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW200529623A (en) * | 2004-01-14 | 2005-09-01 | Nec Corp | Communication encryption method, communication encryption system, terminal device, DNS server and program |
| CN101706805B (en) * | 2009-10-30 | 2011-11-09 | 中国科学院计算技术研究所 | Method and system for storing object |
| CN101714064B (en) * | 2009-12-29 | 2012-01-04 | 成都市华为赛门铁克科技有限公司 | Data access method and server |
| CN103236985A (en) * | 2013-04-02 | 2013-08-07 | 浪潮电子信息产业股份有限公司 | Multipath load balancing system for accessing storage |
| CN103324444B (en) * | 2013-05-24 | 2017-09-22 | 浪潮电子信息产业股份有限公司 | A kind of many control I O scheduling methods that host side is cooperateed with storage end |
| CN104424251B (en) * | 2013-08-28 | 2019-03-15 | 腾讯科技（深圳）有限公司 | A kind of calculation method and system that various dimensions are split |
-
2016
- 2016-04-14 CN CN201610232333.6A patent/CN107301090B/en active Active
-
2017
- 2017-04-13 WO PCT/CN2017/080480 patent/WO2017177953A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| WO2017177953A1 (en) | 2017-10-19 |
| CN107301090A (en) | 2017-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7138126B2 (en) | Timeliness resource migration to optimize resource placement | |
| US10503650B2 (en) | Page cache write logging at block-based storage | |
| CN107301090B (en) | Method and device for setting data processing path for application in storage server | |
| US10896102B2 (en) | Implementing secure communication in a distributed computing system | |
| JP2020024733A (en) | Dynamic configuration of data volumes | |
| JP6381801B2 (en) | Dynamic scaling of storage volume for storage client file system | |
| US8458287B2 (en) | Erasure coded storage aggregation in data centers | |
| US9342457B2 (en) | Dynamically modifying durability properties for individual data volumes | |
| WO2018059222A1 (en) | File slice uploading method and apparatus, and cloud storage system | |
| US10505862B1 (en) | Optimizing for infrastructure diversity constraints in resource placement | |
| US10616134B1 (en) | Prioritizing resource hosts for resource placement | |
| US11204702B2 (en) | Storage domain growth management | |
| CN107423301B (en) | Data processing method, related equipment and storage system | |
| CN110134338B (en) | Distributed storage system and data redundancy protection method and related equipment thereof | |
| CN112948279A (en) | Method, apparatus and program product for managing access requests in a storage system | |
| Hsieh et al. | The incremental load balance cloud algorithm by using dynamic data deployment | |
| US20070233449A1 (en) | Simulation of hierarchical storage systems | |
| US10594620B1 (en) | Bit vector analysis for resource placement in a distributed system | |
| CN112631994A (en) | Data migration method and system | |
| US10812408B1 (en) | Preventing concentrated selection of resource hosts for placing resources | |
| US11068192B1 (en) | Utilizing mutiple snapshot sources for creating new copy of volume in a networked environment wherein additional snapshot sources are reserved with lower performance levels than a primary snapshot source | |
| CN114116655A (en) | Parallel file storage method, device, equipment and storage medium based on virtual machine | |
| US11336519B1 (en) | Evaluating placement configurations for distributed resource placement | |
| US11061835B1 (en) | Sensitivity matrix for system load indication and overload prevention | |
| WO2017122263A1 (en) | Management computer and management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |