CN115878319A - Load balancing method and system and electronic equipment - Google Patents
Load balancing method and system and electronic equipment Download PDFInfo
- Publication number
- CN115878319A CN115878319A CN202211601188.6A CN202211601188A CN115878319A CN 115878319 A CN115878319 A CN 115878319A CN 202211601188 A CN202211601188 A CN 202211601188A CN 115878319 A CN115878319 A CN 115878319A
- Authority
- CN
- China
- Prior art keywords
- data server
- feature vector
- matrix
- feature
- server node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application relates to the field of data server nodes, and particularly discloses a load balancing method, a load balancing system and electronic equipment. And calculating the hyperbolic distance between two feature vectors based on a distance formula between any two points of a Peng Jialai planar model or a Peng Jialai disk model, so as to facilitate describing the geometry of the network. In this way, whether the load of each data server node is balanced can be determined, so that the access performance of the service is ensured.
Description
Technical Field
The present invention relates to the field of data server nodes, and more particularly, to a load balancing method, system and electronic device.
Background
With the continuous development of internet technology, the appearance of a distributed file system effectively solves the difficult problems of data storage and management. A distributed file system generally consists of a metadata server that stores metadata for the distributed file system and a data server that stores actual user data for the distributed file system.
In a distributed file system, a user may add or delete data server nodes dynamically according to the increase or decrease of the data volume, so that the total number of data servers may change, and at the same time, the data server nodes may be dynamically offline or online due to the conditions of node disk damage and the like during the use process, and these conditions may cause imbalance in data distribution and access volume among the data server nodes, and may also cause imbalance in data server node load, thereby affecting the access performance of the service. Therefore, in order to keep the load of the data server nodes in a balanced state, a load balancing scheme is desired.
Disclosure of Invention
The present application is proposed to solve the above-mentioned technical problems. Embodiments of the present application provide a load balancing method, system and electronic device, which employ a convolutional neural network based on deep learning to mine high-dimensional correlation characteristics of disk usage of each data server node in two dimensions, namely a node sample dimension and a time dimension. And calculating the hyperbolic distance between two feature vectors based on a distance formula between any two points of a Peng Jialai planar model or a Peng Jialai disk model, so as to facilitate describing the geometry of the network. In this way, whether the load of each data server node is balanced can be determined, so that the access performance of the service is ensured.
According to an aspect of the present application, there is provided a load balancing method, including: acquiring historical data of disk usage of each data server node; constructing historical data of the disk usage of each data server node into a historical data matrix, wherein the rows and columns of the historical data matrix respectively correspond to the node sample dimension and the time dimension of the data server node; obtaining a usage characteristic diagram of high-dimensional correlation characteristics of the disk usage of each data server node in a node sample dimension and a time dimension by using a convolutional neural network from the historical data matrix, wherein the size of the usage characteristic diagram is S X T C, S is the sample dimension, T is the time dimension, and C is the channel dimension; calculating a global mean value of each feature matrix of the usage feature graph in a time dimension to obtain a first feature vector, wherein the first feature vector is used for representing a time high-dimensional change feature of the disk usage of each data server node in the global direction; calculating the average value of the global normal vectors of the usage characteristic diagram along the channel dimension to obtain a characteristic matrix with the dimension S x T; obtaining the current disk usage of each data server node and forming the current disk usage of each data server node into a current vector; encoding the current vector into a second feature vector through an encoder, wherein the length of the second feature vector is S; multiplying the second eigenvector by the eigenvector matrix, and mapping the current disk usage of each data server node into a time-high-dimensional changed eigenspace to obtain a third eigenvector with the length of T; respectively calculating hyperbolic distances between the first feature vector and the third feature vector to obtain a fourth feature vector with the length of T; taking the fourth feature vector as a query vector to be subjected to matrix multiplication with the feature matrix to obtain a fifth feature vector with the length of S; calculating Softmax-like function values of all positions in the fifth feature vector to obtain probability values of whether the load of each data server node is balanced; and determining whether the load of each data server node is balanced or not based on the Softmax-like function value.
According to another aspect of the present application, there is provided a load balancing system, comprising: the historical data acquisition unit is used for acquiring the historical data of the disk usage of each data server node; the matrix construction unit is used for constructing the historical data of the disk usage of each data server node, which is obtained by the historical data acquisition unit, into a historical data matrix, wherein the rows and the columns of the historical data matrix respectively correspond to the node sample dimension and the time dimension of the data server node; a convolutional neural network processing unit, configured to obtain, by using a convolutional neural network, a usage characteristic map of high-dimensional correlation characteristics of disk usage of each data server node in a node sample dimension and a time dimension from the historical data matrix obtained by the matrix construction unit, where a scale of the usage characteristic map is S × T × C, where S is a sample dimension, T is a time dimension, and C is a channel dimension; a first feature vector generation unit, configured to calculate a global average value of each feature matrix of the usage feature map obtained by the convolutional neural network processing unit in a time dimension to obtain a first feature vector, where the first feature vector is used to represent a time high-dimensional variation feature of the disk usage of each data server node in the global; the characteristic matrix generating unit is used for calculating the average value of the global normal vectors of the usage characteristic diagram along the channel dimension, which is obtained by the convolutional neural network processing unit, so as to obtain a characteristic matrix with the dimension of S x T; the current vector construction unit is used for acquiring the current disk usage of each data server node and forming the current disk usage of each data server node into a current vector; the encoding unit is used for encoding the current vector obtained by the current vector construction unit into a second feature vector through an encoder, and the length of the second feature vector is S; a third eigenvector generating unit, configured to perform matrix multiplication on the second eigenvector obtained by the encoding unit and the eigenvector obtained by the eigenvector generating unit, and map the current disk usage of each data server node into an eigenvector space with a time-high dimensional change, so as to obtain a third eigenvector with a length T; a hyperbolic distance calculating unit, configured to calculate hyperbolic distances between the first feature vector obtained by the first feature vector generating unit and the third feature vector obtained by the third feature vector generating unit, respectively, so as to obtain a fourth feature vector with a length T; a fifth feature vector generating unit, configured to perform matrix multiplication on the fourth feature vector obtained by the hyperbolic distance calculating unit as a query vector and the feature matrix obtained by the feature matrix generating unit to obtain a fifth feature vector with a length of S; a probability value calculation unit, configured to calculate a Softmax-like function value of each position in the fifth feature vector obtained by the fifth feature vector generation unit, so as to obtain a probability value of whether the load of each data server node is balanced; and a result determining unit configured to determine whether the load of each of the data server nodes is balanced based on the Softmax-like function value obtained by the probability value calculating unit.
According to yet another aspect of the present application, there is provided an electronic device including: a processor; and a memory having stored therein computer program instructions which, when executed by the processor, cause the processor to perform the load balancing method as described above.
According to yet another aspect of the present application, there is provided a computer readable medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform a load balancing method as described above.
Compared with the prior art, the load balancing method, the load balancing system and the electronic equipment provided by the application adopt the convolutional neural network based on deep learning to mine the high-dimensional correlation characteristics of the disk usage of each data server node in two dimensions, namely the node sample dimension and the time dimension. And calculating the hyperbolic distance between two feature vectors based on a distance formula between any two points of a Peng Jialai planar model or a Peng Jialai disk model, so as to facilitate describing the geometry of the network. In this way, it can be determined whether the load of each data server node is balanced, thereby ensuring the access performance of the service.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally represent like parts or steps.
Fig. 1 is an application scenario diagram of a load balancing method according to an embodiment of the present application;
FIG. 2 is a flow chart of a load balancing method according to an embodiment of the present application;
fig. 3 is a schematic system architecture diagram of a load balancing method according to an embodiment of the present application;
FIG. 4 is a block diagram of a load balancing system according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device according to an embodiment of the application.
Detailed Description
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and that the present application is not limited by the example embodiments described herein.
Overview of a scene
As described above, in the distributed file system, a user may add or delete data server nodes dynamically according to the increase or decrease of the data volume, so that the total number of data servers may change, and at the same time, the data server nodes may be dynamically offline or online due to the damage of the node disk during the use process, which may cause the data distribution and the access volume between the data server nodes to be unbalanced, and thus may cause the load of the data server nodes to be unbalanced, thereby affecting the access performance of the service. Therefore, in order to keep the load of the data server nodes in a balanced state, a load balancing scheme is desired.
Based on this, in the technical solution of the present application, historical data of disk usage of each data server node is first obtained, and a historical data matrix is constructed based on the historical data, that is, rows and columns of the historical data matrix respectively correspond to a node sample dimension and a time dimension of the data server node. And then, inputting the historical data matrix into a convolutional neural network to obtain a usage characteristic map, and thus mining high-dimensional correlation characteristics of the disk usage of each data server node in two dimensions, namely a node sample dimension and a time dimension.
Here, the dimension of the dose feature map is S × T × C, where S is a sample dimension, T is a time dimension, and C is a channel dimension, in order to more clearly represent the change in the time dimension, a global mean value of the feature matrix of each S × C may be calculated to obtain a first feature vector, i.e., a hidden vector along the time dimension, for expressing the time-high-dimensional change feature in the global.
And then, acquiring the current disk usage of each data server node, forming a current vector, and mapping the current vector to a high-dimensional feature space through an encoder to obtain a second feature vector. And calculating the average value of the global normal vectors along the channel dimension by using the quantity characteristic diagram to obtain a characteristic matrix of S x T, and continuously multiplying the characteristic matrix by the second characteristic vector to obtain a third characteristic vector. Here, the length of the third feature vector is also T, so that the data of the current disk usage is also converted into the feature space of the temporal high-dimensional variation feature.
It is noted that, since the third eigenvector and the first eigenvector are both globally pooled in other dimensions to emphasize the time dimension, each eigenvalue of the third eigenvector and the first eigenvector is actually a representation of an area or a volume within the eigenspace in the eigenspace of the corresponding dose eigenmap. Considering that in the hyperbolic space, the growth rate of the area and the volume relative to the radius is much faster than that in the traditional euclidean space, so that the description of the geometry of the network is more convenient in the hyperbolic space, the hyperbolic distance between the feature values of each corresponding position of the first feature vector and the third feature vector is calculated separately, for example, the hyperbolic distance based on the formula of the distance between any two points of the Peng Jialai planar model or the Peng Jialai disk model, so as to obtain the fourth feature vector. Here, the length of the fourth feature vector is also T.
And finally, taking the fourth feature vector as a query vector, and multiplying the query vector by the feature matrix of S x T to obtain a fifth feature vector with the length of S, wherein each position of the fifth feature vector corresponds to a node sample, namely a data server node. In this way, by calculating the Softmax-like function value of the fifth feature vector, a probability value of whether the load of each data server node is balanced can be obtained.
Based on this, the present application provides a load balancing method, which includes: acquiring historical data of disk usage of each data server node; constructing historical data of the disk usage of each data server node into a historical data matrix, wherein the rows and columns of the historical data matrix respectively correspond to the node sample dimension and the time dimension of the data server node; obtaining a usage characteristic diagram of high-dimensional correlation characteristics of the disk usage of each data server node in a node sample dimension and a time dimension from the historical data matrix by using a convolutional neural network, wherein the size of the usage characteristic diagram is S x T C, S is the sample dimension, T is the time dimension, and C is the channel dimension; calculating a global mean value of each feature matrix of the usage feature map in a time dimension to obtain a first feature vector, wherein the first feature vector is used for representing a time high-dimensional change feature of the disk usage of each data server node in the global; calculating the average value of the global normal vectors of the usage characteristic diagram along the channel dimension to obtain a characteristic matrix with the dimension S x T; obtaining the current disk usage of each data server node and forming the current disk usage of each data server node into a current vector; encoding the current vector into a second feature vector through an encoder, wherein the length of the second feature vector is S; multiplying the second eigenvector by the eigenvector matrix, and mapping the current disk usage of each data server node into a time-high-dimensional changed eigenspace to obtain a third eigenvector with the length of T; respectively calculating hyperbolic distances between the first feature vector and the third feature vector to obtain a fourth feature vector with the length of T; taking the fourth feature vector as a query vector to be subjected to matrix multiplication with the feature matrix to obtain a fifth feature vector with the length of S; calculating Softmax-like function values of all positions in the fifth feature vector to obtain probability values of whether the load of each data server node is balanced; and determining whether the load of each data server node is balanced or not based on the Softmax-like function value.
Fig. 1 illustrates an application scenario diagram of a load balancing method according to an embodiment of the present application. As shown in fig. 1, in the application scenario, first, historical data of disk usage of each data server node (e.g., P as illustrated in fig. 1) is obtained from a cloud (e.g., T as illustrated in fig. 1), and current disk usage of each data server node is obtained. Then, the obtained historical data and current data of the disk usage of each data server node are input into a server (e.g., S as illustrated in fig. 1) deployed with a load balancing algorithm, where the server can process the historical data and the current data of the disk usage of each data server node with the load balancing algorithm to generate a probability value representing whether the load of each data server node is balanced. Further, determining whether the load of each data server node is balanced or not based on the probability value, specifically, determining that the load of the corresponding data server node is unbalanced when the probability value is greater than or equal to a preset threshold value; and when the probability value is less than or equal to a preset threshold value, determining the load balance of the corresponding data server node. In this way, the access performance of the service can be guaranteed.
Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described with reference to the accompanying drawings.
Exemplary method
Fig. 2 illustrates a flow chart of a method of load balancing. As shown in fig. 2, a load balancing method according to an embodiment of the present application includes: s110, acquiring historical data of the disk usage of each data server node; s120, constructing historical data of the disk usage of each data server node into a historical data matrix, wherein the rows and columns of the historical data matrix respectively correspond to the node sample dimension and the time dimension of the data server node; s130, obtaining a usage characteristic graph of high-dimensional correlation characteristics of the disk usage of each data server node in a node sample dimension and a time dimension from the historical data matrix by using a convolutional neural network, wherein the scale of the usage characteristic graph is S X T C, S is the sample dimension, T is the time dimension, and C is the channel dimension; s140, calculating a global mean value of each feature matrix of the usage feature map in a time dimension to obtain a first feature vector, wherein the first feature vector is used for representing a time high-dimensional change feature of the disk usage of each data server node in the global direction; s150, calculating the average value of the global normal vectors of the usage characteristic diagram along the channel dimension to obtain a characteristic matrix with the dimension of S x T; s160, obtaining the current disk usage of each data server node and forming the current disk usage of each data server node into a current vector; s170, encoding the current vector into a second feature vector through an encoder, wherein the length of the second feature vector is S; s180, multiplying the second eigenvector by the eigenvector matrix, and mapping the current disk usage of each data server node into a time-high-dimensional changed eigenvector space to obtain a third eigenvector with a length of T; s190, hyperbolic distances between the first feature vector and the third feature vector are calculated respectively to obtain a fourth feature vector with the length of T; s200, taking the fourth feature vector as a query vector to be subjected to matrix multiplication with the feature matrix to obtain a fifth feature vector with the length of S; s210, calculating Softmax-like function values of all positions in the fifth feature vector to obtain probability values of whether the load of each data server node is balanced; and S220, determining whether the load of each data server node is balanced or not based on the Softmax-like function value.
Fig. 3 illustrates an architecture diagram of a load balancing method according to an embodiment of the present application. As shown in fig. 3, in the network architecture of the load balancing method, first, the obtained historical data (e.g., P as illustrated in fig. 3) of the disk usage of each data server node is constructed as a historical data matrix (e.g., M1 as illustrated in fig. 3); then, obtaining a usage feature map (e.g., F1 as illustrated in fig. 3) for representing high-dimensional correlation features of disk usage of each data server node in a node sample dimension and a time dimension from the historical data matrix using a convolutional neural network (e.g., CNN as illustrated in fig. 3); then, calculating a global mean of each feature matrix of the usage feature map in a time dimension to obtain a first feature vector (e.g., VF1 as illustrated in fig. 3); next, calculating the mean of the global normal vectors along the channel dimensions of the usage signatures to obtain a signature matrix (e.g., MF as illustrated in fig. 3) with a scale S × T; then, the obtained current disk usage (e.g., Q as illustrated in fig. 3) of each of the data server nodes is made up into a current vector (e.g., V1 as illustrated in fig. 3); then, the current vector is encoded into a second feature vector (e.g., VF2 as illustrated in fig. 3) by an encoder (e.g., as illustrated in fig. 3); then, performing matrix multiplication on the second eigenvector and the eigenvector matrix to map the current disk usage of each data server node into an eigenvector space with high dimensional change in time, so as to obtain a third eigenvector (for example, VF3 as illustrated in fig. 3) with a length T; then, hyperbolic distances between the first feature vector and the third feature vector are respectively calculated to obtain a fourth feature vector with a length T (e.g., VF4 as illustrated in fig. 3); then, matrix-multiplying the fourth eigenvector as a query vector with the eigenvector matrix to obtain a fifth eigenvector with length S (e.g., VF5 as illustrated in fig. 3); then, calculating Softmax-like function values for respective locations in the fifth eigenvector to obtain probability values (e.g., PV as illustrated in FIG. 3) of whether the load of each of the data server nodes is balanced; and finally, determining whether the load of each data server node is balanced or not based on the Softmax-like function value.
In step S110 and step S120, historical data of disk usage of each data server node is obtained, and the historical data of the disk usage of each data server node is constructed as a historical data matrix, where rows and columns of the historical data matrix correspond to a node sample dimension and a time dimension of the data server node, respectively. It should be understood that, considering that there is an association between the current disk usage of each data server node in the distributed file system, such implicit association features may be mined based on historical data of the current disk usage of each of the data server nodes in the present application. Therefore, in the technical solution of the present application, it is first required to obtain historical data of disk usage of each data server node, and construct a historical data matrix based on the historical data, that is, rows and columns of the historical data matrix respectively correspond to a node sample dimension and a time dimension of the data server node. In a specific example, historical data of disk usage of each data server node may be obtained from a cloud.
Specifically, in the embodiment of the present application, constructing the historical data of the disk usage of each data server node as a historical data matrix includes: firstly, arranging the disk usage of each data server node at each time point into vectors; then, the vectors corresponding to the data server nodes are arranged in rows to obtain the historical data matrix.
In step S130, a usage characteristic map of a high-dimensional correlation characteristic of the disk usage of each data server node in a node sample dimension and a time dimension is obtained from the historical data matrix using a convolutional neural network, where a scale of the usage characteristic map is S × T × C, where S is the sample dimension, T is the time dimension, and C is the channel dimension. That is, the historical data matrix is input into a convolutional neural network for processing, so as to mine high-dimensional associated features of the disk usage of each data server node in two dimensions, namely a node sample dimension and a time dimension, and thus a usage feature map is obtained. It is worth mentioning that, here, the scale of the usage feature graph is S × T × C, where S is a sample dimension, T is a time dimension, and C is a channel dimension, and the usage feature graph is a correlation feature that merges the disk usage of each data server node between the sample dimension and the time dimension of the node.
In step S140, a global mean of each feature matrix of the usage feature map in the time dimension is calculated to obtain a first feature vector, where the first feature vector is used to represent a time high-dimensional variation feature of the disk usage of each data server node in the global. It should be understood that, in order to be able to more clearly represent the change in the time dimension, in the technical solution of the present application, a global mean of the feature matrix of each S × C in the usage feature map may be calculated to obtain a first feature vector, that is, a hidden vector along the time dimension, so as to represent the time high-dimensional change feature of the disk usage of each data server node in the global.
In step S150, the mean of the global normal vectors along the channel dimensions of the dose feature map is calculated to obtain a feature matrix with a dimension S × T. It should be understood that, in order to facilitate subsequent fusion with the data high-dimensional feature of the current disk usage of each data server node, the data of the current disk usage is converted into the feature space of the temporal high-dimensional change feature. Therefore, in the technical solution of the present application, the usage feature map is averaged along a global normal vector of a channel dimension to obtain a feature matrix with a dimension S × T.
In step S160 and step S170, the current disk usage of each data server node is obtained, the current disk usage of each data server node is formed into a current vector, and the current vector is encoded into a second feature vector by an encoder, where the length of the second feature vector is S. It should be understood that, in order to extract the relevance feature between the current disk usage amounts of the server nodes, in the technical solution of the present application, after processing the historical data of the current disk usage amounts of the data server nodes, feature extraction needs to be performed on the relevance feature between the current disk usage amounts of the data server nodes. That is, first, the current disk usage of each data server node is obtained; then, constructing the obtained current disk usage as a current vector; and finally, processing the current vector through an encoder to map the current vector into a high-dimensional space, so as to facilitate subsequent calculation and classification, thereby obtaining a second feature vector.
Specifically, in this embodiment of the present application, the process of encoding the current vector into a second feature vector by an encoder, where the length of the second feature vector is S, includes: firstly, extracting high-dimensional implicit features of feature values of all positions in the current vector by using a full-connection layer of the encoder. Then, a high-dimensional implicit feature of the association between the feature values of the respective positions in the current vector is extracted using the one-dimensional convolutional layer of the encoder to obtain the second feature vector.
In step S180, matrix-multiplying the second eigenvector and the eigenvector matrix, and mapping the current disk usage of each data server node into an eigenspace with a time-high dimensional change, so as to obtain a third eigenvector with a length T. It should be understood that the scale of the feature matrix is S × T, and the length of the second feature vector is S, so in the technical solution of the present application, a third feature vector with a length T is obtained after matrix-multiplying the second feature vector and the feature matrix. That is, the second eigenvector is matrix-multiplied with the eigenvector matrix to map the current disk usage of each of the data server nodes into an eigenspace of a temporal high-dimensional change, thereby obtaining the third eigenvector. This enables both the first feature vector and the third feature vector to appear in a feature space emphasizing a time dimension for subsequent processing thereof.
In step S190, hyperbolic distances between the first feature vector and the third feature vector are respectively calculated to obtain a fourth feature vector with a length T. It should be appreciated that since the third eigenvector and the first eigenvector are both globally pooled in other dimensions to emphasize the temporal dimension, each eigenvalue of the third eigenvector and the first eigenvector is actually a representation of an area or volume within the eigenspace of the corresponding usage eigenmap. And considering that the area and the volume increase speed relative to the radius in the hyperbolic space is much faster than that in the traditional Euclidean space, so that the geometrical structure of the network is more conveniently described in the hyperbolic space. Therefore, in the technical solution of the present application, hyperbolic distances between feature values of each corresponding position of the first feature vector and the third feature vector are respectively calculated, and in a specific example, hyperbolic distances based on a distance formula between any two points of a Peng Jialai planar model or a Peng Jialai disk model may be calculated, so as to obtain a fourth feature vector. Here, the length of the fourth feature vector is also T.
In step S200, matrix-multiplying the fourth feature vector as a query vector with the feature matrix to obtain a fifth feature vector with a length S. That is, the fourth feature vector is used as a query vector to be matrix-multiplied by the feature matrix with the scale S × T, so as to obtain a fifth feature vector with the length S. It should be noted that, here, the fifth feature vector represents a high-dimensional associated feature that fuses associated information between the fourth feature vector and the feature matrix, and each position in the fifth feature vector corresponds to a node sample, that is, a data server node.
In steps S210 and S220, a Softmax-like function value of each position in the fifth feature vector is calculated to obtain a probability value of whether the load of each of the data server nodes is balanced, and it is determined whether the load of each of the data server nodes is balanced based on the Softmax-like function value. That is, in one specific example, first, the Softmax-like function value of each position in the fifth feature vector is calculated by the following formula: y = exp (pi)/∑ s i exp (pi), where y is the Softmax-like function value, and pi is the eigenvalue of each position in the fifth eigenvector, so that a probability value of whether the load of each data server node is balanced can be obtained. Then, based on the Softmax-like function value, determiningWhether the load of each of the data server nodes is balanced.
Specifically, in this embodiment of the present application, the determining, based on the Softmax-like function value, whether the load of each data server node is balanced includes: firstly, in response to the fact that the Softmax-like function value is larger than or equal to a preset threshold value, determining load imbalance of the corresponding data server node; then, in response to the Softmax-like function value being less than or equal to a preset threshold, determining load balancing of the corresponding data server node.
In summary, the load balancing method of the embodiment of the present application is illustrated, which employs a convolutional neural network based on deep learning to mine high-dimensional correlation characteristics of the disk usage of each data server node in two dimensions, namely a node sample dimension and a time dimension. And calculating the hyperbolic distance between two feature vectors based on a distance formula between any two points of a Peng Jialai planar model or a Peng Jialai disk model, so as to facilitate describing the geometry of the network. In this way, it can be determined whether the load of each data server node is balanced, thereby ensuring the access performance of the service.
Exemplary System
FIG. 4 illustrates a block diagram of a load balancing system according to an embodiment of the present application. As shown in fig. 4, a load balancing system 400 according to an embodiment of the present application includes: a history data obtaining unit 410, configured to obtain history data of disk usage of each data server node; a matrix constructing unit 420, configured to construct the historical data of the disk usage of each data server node, which is obtained by the historical data obtaining unit 410, into a historical data matrix, where rows and columns of the historical data matrix respectively correspond to a node sample dimension and a time dimension of the data server node; a convolutional neural network processing unit 430, configured to obtain, by using a convolutional neural network, a usage feature map of high-dimensional correlation features of disk usage of each data server node in a node sample dimension and a time dimension from the historical data matrix obtained by the matrix constructing unit 420, where the scale of the usage feature map is S × T × C, where S is a sample dimension, T is a time dimension, and C is a channel dimension; a first feature vector generation unit 440, configured to calculate a global average value of each feature matrix of the usage feature map obtained by the convolutional neural network processing unit 430 in a time dimension to obtain a first feature vector, where the first feature vector is used to represent a time high-dimensional variation feature of the disk usage of each data server node in the global; a feature matrix generating unit 450, configured to calculate a mean value of global normal vectors along a channel dimension of the usage feature map obtained by the convolutional neural network processing unit 430 to obtain a feature matrix with a scale S × T; a current vector constructing unit 460, configured to obtain current disk usage of each data server node and form the current disk usage of each data server node into a current vector; an encoding unit 470, configured to encode the current vector obtained by the current vector constructing unit 460 into a second feature vector through an encoder, where the length of the second feature vector is S; a third eigenvector generating unit 480, configured to perform matrix multiplication on the second eigenvector obtained by the encoding unit 470 and the eigenvector obtained by the eigenvector generating unit 450, and map the current disk usage of each data server node into an eigenvector space with a time-high dimensional change, so as to obtain a third eigenvector with a length T; a hyperbolic distance calculating unit 490, configured to calculate hyperbolic distances between the first feature vector obtained by the first feature vector generating unit 440 and the third feature vector obtained by the third feature vector generating unit 480 respectively to obtain a fourth feature vector with a length T; a fifth feature vector generating unit 500, configured to perform matrix multiplication on the fourth feature vector obtained by the hyperbolic distance calculating unit 490 as a query vector and the feature matrix obtained by the feature matrix generating unit 450 to obtain a fifth feature vector with a length S; a probability value calculation unit 510, configured to calculate Softmax-like function values of respective positions in the fifth eigenvector obtained by the fifth eigenvector generation unit 500, so as to obtain a probability value of whether the load of each data server node is balanced; and a result determining unit 520, configured to determine whether the load of each data server node is balanced based on the Softmax-like function value obtained by the probability value calculating unit 510.
In one example, in the load balancing system 400, the matrix constructing unit 420 is further configured to: arranging the disk usage of each data server node at each time point into vectors; and arranging vectors corresponding to the data server nodes according to rows to obtain the historical data matrix.
In one example, in the load balancing system 400, the encoding unit 470 includes: a full-connection processing subunit, configured to extract high-dimensional implicit features of feature values at each position in the current vector using a full-connection layer of the encoder; and the one-dimensional convolution processing subunit is used for extracting the associated high-dimensional implicit features between the feature values of all the positions in the current vector by using the one-dimensional convolution layer of the encoder so as to obtain the second feature vector.
In one example, in the load balancing system 400 described above, the hyperbolic distance is a hyperbolic distance based on any two-point distance formula of a Peng Jialai planar model or a Peng Jialai disk model.
In an example, in the above load balancing system 400, the probability value calculating unit 510 is further configured to: calculating a Softmax-like function value of each position in the fifth feature vector by using the following formula: y = exp (pi)/∑ i exp (pi), where y is the Softmax-like function value and pi is the eigenvalue of each position in the fifth eigenvector.
In one example, in the load balancing system 400, the result determining unit 520 includes: a respective greater-than subunit to determine a load imbalance for the respective data server node in response to the Softmax-like function value being greater than or equal to a preset threshold; and a respective sub-unit, configured to determine load balancing for the respective data server node in response to the Softmax-like function value being less than or equal to a preset threshold.
Here, it will be understood by those skilled in the art that the specific functions and operations of the respective units and modules in the load balancing system 400 described above have been described in detail in the description of the load balancing method above with reference to fig. 1 to 3, and thus, a repetitive description thereof will be omitted.
As described above, the load balancing system 400 according to the embodiment of the present application may be implemented in various terminal devices, such as a server of a load balancing algorithm. In one example, the load balancing system 400 according to the embodiment of the present application may be integrated into the terminal device as a software module and/or a hardware module. For example, the load balancing system 400 may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, the load balancing system 400 may also be one of many hardware modules of the terminal device.
Alternatively, in another example, the load balancing system 400 and the terminal device may be separate devices, and the load balancing system 400 may be connected to the terminal device through a wired and/or wireless network and transmit the interaction information according to the agreed data format.
Exemplary electronic device
Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 5. As shown in fig. 5, the electronic device 10 includes one or more processors 11 and memory 12. The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
In one example, the electronic device 10 may further include: an input system 13 and an output system 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
The input system 13 may comprise, for example, a keyboard, a mouse, etc.
The output system 14 may output various information including probability values and the like to the outside. The output system 14 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 10 relevant to the present application are shown in fig. 5, and components such as buses, input/output interfaces, and the like are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer-readable storage Medium
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform steps in functions of a load balancing method according to various embodiments of the present application described in the "exemplary methods" section above of this specification.
The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the load balancing method described in the "exemplary methods" section above in this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present application have been described above with reference to specific embodiments, but it should be noted that advantages, effects, etc. mentioned in the present application are only examples and are not limiting, and the advantages, effects, etc. must not be considered to be possessed by various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is provided for purposes of illustration and understanding only, and is not intended to limit the application to the details which are set forth in order to provide a thorough understanding of the present application.
The block diagrams of devices, apparatuses, devices, systems referred to in this application are only used as illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (10)
1. A method of load balancing, comprising: acquiring historical data of disk usage of each data server node; constructing historical data of the disk usage of each data server node into a historical data matrix, wherein the rows and columns of the historical data matrix respectively correspond to the node sample dimension and the time dimension of the data server node; obtaining a usage characteristic diagram of high-dimensional correlation characteristics of the disk usage of each data server node in a node sample dimension and a time dimension by using a convolutional neural network from the historical data matrix, wherein the size of the usage characteristic diagram is S X T C, S is the sample dimension, T is the time dimension, and C is the channel dimension; calculating a global mean value of each feature matrix of the usage feature graph in a time dimension to obtain a first feature vector, wherein the first feature vector is used for representing a time high-dimensional change feature of the disk usage of each data server node in the global direction; calculating the average value of the global normal vectors of the usage characteristic diagram along the channel dimension to obtain a characteristic matrix with the dimension S x T; obtaining the current disk usage of each data server node and forming the current disk usage of each data server node into a current vector; encoding the current vector into a second feature vector through an encoder, wherein the length of the second feature vector is S; multiplying the second eigenvector by the eigenvector matrix, and mapping the current disk usage of each data server node into a time-high-dimensional changed eigenspace to obtain a third eigenvector with the length of T; respectively calculating hyperbolic distances between the first feature vector and the third feature vector to obtain a fourth feature vector with the length of T; taking the fourth feature vector as a query vector to be subjected to matrix multiplication with the feature matrix to obtain a fifth feature vector with the length of S; calculating Softmax-like function values of all positions in the fifth feature vector to obtain probability values of whether the load of each data server node is balanced; and determining whether the load of each data server node is balanced or not based on the Softmax-like function value.
2. The load balancing method according to claim 1, wherein constructing historical data of disk usage of each data server node as a historical data matrix comprises: arranging the disk usage of each data server node at each time point into a vector; and arranging vectors corresponding to the data server nodes according to rows to obtain the historical data matrix.
3. The load balancing method of claim 1, wherein encoding the current vector into a second eigenvector of length S by an encoder comprises: extracting high-dimensional implicit features of the feature values of all positions in the current vector by using a full-connection layer of the encoder; and extracting associated high-dimensional implicit features between feature values of all positions in the current vector by using the one-dimensional convolutional layer of the encoder to obtain the second feature vector.
4. The load balancing method according to claim 1, wherein the hyperbolic distance is a hyperbolic distance based on any two-point distance formula of a Peng Jialai planar model or a Peng Jialai disk model.
5. The load balancing method of claim 1, wherein calculating the Softmax-like function value for each position in the fifth eigenvector to obtain a probability value of whether the load of each of the data server nodes is balanced comprises: calculating a Softmax-like function value of each position in the fifth feature vector by using the following formula: y = exp (pi)/∑ i exp (pi), where y is the Softmax-like function value and pi is the eigenvalue of each position in the fifth eigenvector.
6. The load balancing method of claim 1, wherein determining whether the load of each of the data server nodes is balanced based on the class Softmax function value comprises: in response to the Softmax-like function value being greater than or equal to a preset threshold, determining a load imbalance for the corresponding data server node; and responding to the Softmax-like function value being smaller than or equal to a preset threshold value, and determining load balance of the corresponding data server nodes.
7. A load balancing system, comprising: the historical data acquisition unit is used for acquiring the historical data of the disk consumption of each data server node; the matrix construction unit is used for constructing the historical data of the disk usage of each data server node, which is obtained by the historical data acquisition unit, into a historical data matrix, wherein the rows and the columns of the historical data matrix respectively correspond to the node sample dimension and the time dimension of the data server node; a convolutional neural network processing unit, configured to obtain, by using a convolutional neural network, a usage characteristic map of high-dimensional correlation characteristics of disk usage of each data server node in a node sample dimension and a time dimension from the historical data matrix obtained by the matrix construction unit, where a scale of the usage characteristic map is S × T × C, where S is a sample dimension, T is a time dimension, and C is a channel dimension; a first feature vector generation unit, configured to calculate a global average value of each feature matrix of the usage feature map obtained by the convolutional neural network processing unit in a time dimension to obtain a first feature vector, where the first feature vector is used to represent a time high-dimensional variation feature of the disk usage of each data server node in the global; the characteristic matrix generating unit is used for calculating the average value of the global normal vectors of the usage characteristic diagram along the channel dimension, which is obtained by the convolutional neural network processing unit, so as to obtain a characteristic matrix with the dimension of S x T; the current vector construction unit is used for acquiring the current disk usage of each data server node and forming the current disk usage of each data server node into a current vector; the encoding unit is used for encoding the current vector obtained by the current vector construction unit into a second feature vector through an encoder, and the length of the second feature vector is S; a third eigenvector generating unit, configured to perform matrix multiplication on the second eigenvector obtained by the encoding unit and the eigenvector obtained by the eigenvector generating unit, and map the current disk usage of each data server node into an eigenvector space with a time-high dimensional change, so as to obtain a third eigenvector with a length T; a hyperbolic distance calculating unit, configured to calculate hyperbolic distances between the first feature vector obtained by the first feature vector generating unit and the third feature vector obtained by the third feature vector generating unit, respectively, so as to obtain a fourth feature vector with a length T; a fifth feature vector generating unit, configured to perform matrix multiplication on the fourth feature vector obtained by the hyperbolic distance calculating unit as a query vector and the feature matrix obtained by the feature matrix generating unit to obtain a fifth feature vector with a length of S; a probability value calculation unit configured to calculate a Softmax-like function value of each position in the fifth eigenvector obtained by the fifth eigenvector generation unit to obtain a probability value of whether the load of each data server node is balanced; and a result determining unit configured to determine whether the load of each of the data server nodes is balanced, based on the Softmax-like function value obtained by the probability value calculating unit.
8. The load balancing system of claim 7, wherein the matrix construction unit is further configured to: arranging the disk usage of each data server node at each time point into vectors; and arranging vectors corresponding to the data server nodes according to rows to obtain the historical data matrix.
9. The load balancing system of claim 7, wherein the encoding unit comprises: a full-connection processing subunit, configured to extract high-dimensional implicit features of feature values at each position in the current vector by using a full-connection layer of the encoder; and the one-dimensional convolution processing subunit is used for extracting the associated high-dimensional implicit features between the feature values of all the positions in the current vector by using the one-dimensional convolution layer of the encoder so as to obtain the second feature vector.
10. The load balancing system of claim 7, wherein the probability value calculation unit is further configured to: calculating a Softmax-like function value of each position in the fifth feature vector according to the following formula: y = exp (pi)/∑ i exp (pi), wherein y is the Softma-likeAnd the function value of x, pi is the characteristic value of each position in the fifth characteristic vector.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211601188.6A CN115878319A (en) | 2022-12-13 | 2022-12-13 | Load balancing method and system and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211601188.6A CN115878319A (en) | 2022-12-13 | 2022-12-13 | Load balancing method and system and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115878319A true CN115878319A (en) | 2023-03-31 |
Family
ID=85767365
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211601188.6A Withdrawn CN115878319A (en) | 2022-12-13 | 2022-12-13 | Load balancing method and system and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115878319A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116095089A (en) * | 2023-04-11 | 2023-05-09 | 云南远信科技有限公司 | Remote sensing satellite data processing method and system |
| CN117215766A (en) * | 2023-06-26 | 2023-12-12 | 杭州喜倍科技有限公司 | Multimedia intelligent dispatcher and method thereof |
-
2022
- 2022-12-13 CN CN202211601188.6A patent/CN115878319A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116095089A (en) * | 2023-04-11 | 2023-05-09 | 云南远信科技有限公司 | Remote sensing satellite data processing method and system |
| CN116095089B (en) * | 2023-04-11 | 2023-06-16 | 云南远信科技有限公司 | Remote sensing satellite data processing method and system |
| CN117215766A (en) * | 2023-06-26 | 2023-12-12 | 杭州喜倍科技有限公司 | Multimedia intelligent dispatcher and method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109033068B (en) | Method and device for reading and understanding based on attention mechanism and electronic equipment | |
| CN115878319A (en) | Load balancing method and system and electronic equipment | |
| CN112085565B (en) | Deep learning-based information recommendation method, device, equipment and storage medium | |
| CN114048331A (en) | Knowledge graph recommendation method and system based on improved KGAT model | |
| CN111414987A (en) | Training method and training device for neural network and electronic equipment | |
| CN111178949B (en) | Service resource matching reference data determining method, device, equipment and storage medium | |
| CN115099684B (en) | Enterprise safety production management system and management method thereof | |
| EP4152212A1 (en) | Data processing method and device | |
| CN115759658A (en) | Enterprise energy consumption data management system suitable for smart city | |
| CN111581923A (en) | Method, device and equipment for generating file and computer readable storage medium | |
| CN115373813A (en) | Scheduling method and system based on GPU virtualization in cloud computing environment and electronic equipment | |
| CN115834433A (en) | Data processing method and system based on Internet of things technology | |
| CN116579618B (en) | Data processing method, device, equipment and storage medium based on risk management | |
| CN112070550A (en) | Keyword determination method, device and equipment based on search platform and storage medium | |
| CN115705583A (en) | Multi-target prediction method, device, equipment and storage medium | |
| CN111178986A (en) | User-commodity preference prediction method and system | |
| CN114610488A (en) | Data acquisition method and system based on load balancing and electronic equipment | |
| CN115409617B (en) | Foreign currency exchange method and system based on block chain | |
| CN112800217A (en) | Vector relevance matrix-based intelligent assessment method for government affair transaction processing report | |
| CN110866637B (en) | Scoring prediction method, scoring prediction device, computer equipment and storage medium | |
| JP2021033994A (en) | Text processing method, apparatus, device and computer readable storage medium | |
| CN115186188A (en) | Product recommendation method, device and equipment based on behavior analysis and storage medium | |
| CN114205817A (en) | Wireless local area network access method, system and electronic equipment | |
| CN111459990B (en) | Object processing method, system, computer readable storage medium and computer device | |
| CN116415624A (en) | Model training method and device, and content recommendation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20230331 |
|
| WW01 | Invention patent application withdrawn after publication |